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FOREWORD 


The  Training  and  Siaulation  Technical  Area  (Simulation  Systems  Design 
Team)  of  the  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI)  performs  research  and  development  in  areas  that  include  training  simu¬ 
lation  with  applicability  to  military  training.  Of  special  interest  is  re¬ 
search  in  the  area  of  training  device  design  requirements.  Before  the  Army 
can  develop  and  procure  effective  training  systems,  it  should  first  determine 
the  most  effective  and  efficient  design  for  those  systesms.  However,  adequate 
guidance  for  determining  these  system  designs  is  not  currently  available. 

This  report  provides  the  foundation  for  a  training  system  issues  data 
base.  Such  a  data  base  can  support  user-oriented  guidance  in  the  design  of 
training  systems. 

User-oriented  guidelines  for  making  training  system  design  decisions  will 
facilitate  the  efforts  of  training  device  procurers  such  as  the  Army  Project 
Manager  for  Training  Devices  (PM  TRADE)  and  also  instructional  systems  devel¬ 
opers  in  the  Army  Training  and  Doctrine  Command  (TRADOC). 
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RESEARCH  ISSUES  IN  TRAINING  DEVICE  DESIGN:  THE  ORGANIZATION  OP  A  DATA  BASE 


EXECUTIVE  SUMMARY 


Requi rements : 

To  contribute  to  the  development  o£  empirically  based  guidance  for  the 
design  of  training  devices  and  training  systems  by:  (1)  accumulating  and 
categorizing  the  training  system  research  issues  which  appear  to  be  critical 
in  making  decisions  about  training  device  design,  (2)  conducting  an  initial 
review  and  analysis  of  pertinent  training  device  research,  (3)  proposing  a 
means  for  accumulating  new  and  existing  data,  (4)  proposing  a  means  for  storing 
and  accessing  empirical  information,  and  (5)  proposing  a  method  for  generating 
empirically  derived  training  device  design  guidance. 


Procedure : 

The  paper  is  organized  into  four  main  sections:  an  introduction  and  state¬ 
ment  of  the  problem,  a  discussion  of  how  the  data  derived  from  investigations  of 
training  system  research  issues  may  be  formatted  and  accessed  by  training  devel¬ 
opers  via  a  proposed  expert  system,  an  Instructional  Systems  Development  (ISD) 
based  framework  for  organizing  training  system  research  issues,  and  a  section 
which  defines  the  research  issues  and  reviews  the  literature. 


Findings : 

There  is  a  vital  need  for  empirically  based  guidance  for  the  design  of 
training  devices  and  training  systems.  Current  design  procedures  are  too  reli¬ 
ant  on  subject  matter,  expert  opinion,  or  intuition.  Existing  data  and  new 
data  can  be  accumulated, in to  a  master  data  base  which  may  then  be  used  as  the 
foundation  for  an  expert  system  for  training  device  design  guidance.  The 
accumulation  of  training  system  research  issues  will  serve  as  the  conceptual 
organization  of  the  data  base  and  knowledge  system. 


Utilization  of  Findings: 

This  report  may  be  used  by  researchers  in  planning  for  research  on  train¬ 
ing  system  issues  and  for  the  preliminary  design  of  training  system  design 
guidance  packages. 
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RESEARCH  ISSUES  IN  TRAINING  DEVICE  DESIGN:  THE  ORGANIZATION  OP  A  DATA  BASE 


INTRODUCTION 

In  recent  years  the  Army,  the  other  services,  as  well  as  private  industry, 
have  been  forced  to  reevaluate  the  way  individuals  are  trained.  Economic  con¬ 
straints  have  made  it  unfeasible  to  use  actual  equipment  as  the  backbone  of 
instructional  systems  since  actual  equipment  is  often  needed  in  the  field  and 
cannot  be  spared  to  use  for  training.  In  addition,  new  instructional  techniques 
using  training  devices  and  simulators  have  afforded  the  opportunity  to  train 
tasks  that  are  difficult  or  impossible  to  train  using  only  the  actual  equipment. 
Many  tasks  are  too  dangerous  to  train  on  actual  equipment  or  can  only  be  exer¬ 
cised  in  a  wartime  environment.  The  tasks  themselves  have  also  changed  in 
recent  years.  Hie  use  of  high-technology  equipment  often  enables  easier  opera¬ 
tions  but  demands  more  difficult  maintenance  requirements.  Instructional  sys¬ 
tems  developers  are  now  being  called  upon  to  upgrade  their  maintenance  training 
procedures,  possibly  by  using  more  sophisticated  training  devices  and  simulators. 
If  the  military  is  to  use  more  training  devices  and  simulators  effectively,  it 
must  first  specify  the  required  characteristics  of  these  training  devices. 
Unfortunately,  adequate  guidance  on  how  to  produce  these  specifications  is 
severely  deficient. 

The  process  of  specifying  training  device  characteristics  is  required 
by  several  Army  regulations  (e.g.,  AR  70-1,  AR  71-9)  and  standardized  in  the 
Interservice  Procedures  for  Instructional  Systems  Development  (TRADOC  PAM 
350-30).  Nevertheless,  in  actual  practice,  this  process  is  often  based  on 
intuition  and/or  on  budgetary  constraints.  Subject  matter  experts  (SMEs)  have 
no  empirically  derived  data  base  for  reasonable  guidance  on  these  matters  and 
therefore  often  call  for  high  fidelity  when  they  generate  the  specifications 
for  training  devices.  Based  on  the  recommendations  of  SMEs,  the  individuals 
responsible  for  acquiring  the  training  devices  subsequently  buy  all  the  fidel¬ 
ity  they  can  afford.  There  are  at  least  four  major  problems  with  the  current 
process  of  developing  training  device  requirements  (TDR) :  (1)  There  are 

excessive  amounts  of  different  kinds  of  documentation  in  the  form  of  Army 
Regulations,  Manuals,  technical  reports,  and  guidebooks,  each  purporting  to 
give  the  final  word  in  TDR  development.  (2)  There  is  the  problem  of  maintain¬ 
ing  and  updating  these  documents.  As  new  TDR  information,  regulations,  and 
techniques  are  developed  they  must  be  included  in  the  morass  of  literature 
and  then  extracted  by  the  user.  (3)  Even  with  all  of  this  documentation, 
there  is  no  specific  empirically  based  guidance  for  making  TDR  decisions. 

(4)  These  three  problems  are  exacerbated  for  the  researcher  who  is  attempting 
to  support  the  training  community  by  providing  empirically  based  guidance. 

Not  only  must  the  researcher  determine  where  guidance  is  needed,  but  also  the 
type  of  guidance  must  be  determined.  With  the  current  state  of  knowledge 
there  are  only  a  few  empirically  based  guidance  statements  that  the  researcher 
can  provide.  The  development  of  an  empirically  based  guidance  system  to  help 
in  the  specification  of  training  devices  would  aid  the  decision  processes  of 
instructional  system  developers  and  procurement  officials  as  well  as  improving 
the  cost  effectiveness  of  the  Instructional  Systems  Development  (ISD)  process. 
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The  purpose  of  this  paper  is  to  contribute  to  the  development  of  such  a  j 

guidance  system  by  accumulating  and  categorizing  the  training  system  research  1 

issues  which  appear  to  be  critical  in  making  decisions  about  training  device  1 

design.  In  order  to  avoid  needless  duplication  of  effort,  and  to  take  ad¬ 
vantage  of  the  empirically  generated  information  that  already  exists,  an 
initial  review  and  organization  of  pertinent  training  device  research  is  also 
incorporated  into  this  paper.  In  addition,  a  means  for  accumulating  new  and 
existing  data  and  for  accessing  empirically  derived  training  device  design 
guidance  is  proposed.  Ibis  paper  is  based  on  a  three-faceted  philosophy  of 
training  device  design  and  training  design  development,  Hiis  philosophy  is 
not  new  but  is  believed  to  be  the  most  effective  approach  to  the  development 
of  training  devices  and  systems. 

The  first  facet  of  the  philosophy  is  that  training  needs  and  training 
goals  should  be  established  prior  to  selection  of  any  training  device.  Hie 
specifications  for  any  training  device  should  be  driven  by  training  needs 
and  the  best  instructional  techniques  for  reaching  training  goals.  Hie 
characteristics  of  a  training  device,  no  matter  how  appealing,  should  not 
be  the  determining  influence  in  instructional  design. 

The  second  facet  of  the  philosophy  refers  to  both  the  progression  of 
learning  and  also  the  progression  of  training  device  design  specification.  As 
a  general  approach,  both  progressions  should  proceed  from  simple  to  complex. 

In  learning,  the  trainee  needs  to  move  from  a.  state  of  little  experience 
with  the  task  to  one  of  more  experience  (Glaser,  1982).  Hiis  progression 
can  be  most  efficient  if  the  tasks  to  be  learned  also  move  from  simple  to 
complex.  Hius  the  trainee  should  first  become  familiar  with  the  tools  and 
equipment,  and  then  move  on  to  a  more  complete  understanding  of  its  operation 
(Kinkade  &  Wheaton,  1972;  Pink  6  Shriver,  1978).  In  an  analogous  fashion, 
training  device  designers  should  start  with  simple,  low-fidelity  approaches 
and  only  add  complexity  or  high  fidelity  as  it  is  required  to  improve  training. 

Many  training  device  designs  begin  with  as  high  a  level  of  fidelity  as  possible 
and  then  fidelity  is  reduced  until  a  level  of  affordability  is  reached.  This 
approach  must  be  reoriented  to  move  from  simple  to  complex  training  devices 
only  when  necessary  to  improve  training. 

Finally,  the  third  and  most  important  facet  of  the  philosophy  is  that  all 
ISD  decisions  should  be  both  iterative  and  data  based.  By  iterative,  it  is 
meant  that  no  ISD  decision  should  be  final.  It  should  rather  be  modifiable  as 
more  information  is  obtained  during  the  course  of  the  ISD  process.  By  data 
based,  it  is  meant  that  empirical  data  must  be  a  primary  factor  in  any  training 
device  design  or  training  system  decisions.  Training  system  designers  cannot 
afford  to  rely  on  intuition  or  input  from  subject  natter  experts  who  lack  training 
expertise  in  designing  training  systems.  Hie  establishment  and  use  of  a  data 
base  and  knowledge  system  which  contains  empirical  data  on  all  the  research 
issued  to  be  discussed  and  that  provides  usable  guidance  on  those  issues  will 
be  a  iluable  tool  that  will  Insure  that  the  Army  delivers  the  most  effective 
trai’r  ,ig  possible. 
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Organization 

The  remainder  of  the  paper  is  organized  into  three  main  sections.  In  the 
first  section  we  discuss  how  empirically  derived  data  can  be  formatted  and 
accessed  by  training  device  developers  using  the  proposed  knowledge  base  system. 
In  the  second  section  an  Instructional  Systems  Development  (ISD)  based  frame¬ 
work  for  organizing  training  system  research  issues  is  presented.  This  organiza¬ 
tion  emphasizes  the  concept  that  research  issues  are  not  centered  on  any  single 
variable  or  process  but  rather  must  focus  on  many  interrelated  processes  link¬ 
ing  a  large  variety  of  variables.  In  the  third  section  the  basic  research 
issues  are  defined  and  the  research  literature  on  them  is  reviewed  based  on 
the  ISD  framework  presented  in  the  second  section. 


FORMATTING  AND  ACCESSING  A  TRAINING  DESIGN  GUIDANCE  SYSTEM 

In  the  following  sections  we  will  discuss  many  research  issues  that 
afford  the  opportunity  for  developing  empirically  based  training  device  design 
guidance,  as  well  as  providing  guidance  in  many  of  the  other  areas  of  the  ISD 
process.  In  order  to  make  such  guidance  usable  by  both  the  Army  training 
community  and  the  researchers  who  aid  these  decision  makers,  the  accumulated 
data  must  be  presented  in  a  form  that  is  both  understandable  by  users  and  also 
relatively  easy  for  the  users  to  access.  The  first  criterion  for  usability  of 
a  data  based  training  design  guidance  system,  understandability  by  users, 
involves  the  format  and  organization  of  the  data  base  and  knowledge  system. 

The  second  criterion,  accessibility,  is  mostly  a  question  of  how  the  data  base 
is  packaged  and  distributed  to  the  prospective  users. 


A  Format  for  a  User-Oriented  Data  Base 


Providing  guidance  for  the  decision  maker  in  the  ISD  process  is  the  bottom 
line  goal  of  all  research  on  training  device  and  training  system  design  issues. 
It  is  our  opinion  that  the  most  effective  method  for  meeting  this  goal  is  to 
provide  empirically  based  information  to  the  decision  maker  in  a  form  that  is 
readily  understandable  and  usable  in  the  ISD  decision  process.  This  can  be 
most  efficiently  done  by  organizing  the  existing  information  and  any  new 
empirically  generated  data  in  terms  of  the  decision  steps  involved  in  the 
overall  ISD  process.  To  a  large  degree,  the  decision  questions  discussed  in 
this  article  address  the  decision  points  which  would  be  used  to  organize 
this  proposed  data  base.  The  implication  is  that  the  data  base  and  knowledge 
system  could  thus  be  easily  expanded  or  complemented  by  additional  future 
research  stimulated  by  the  issues  raised  in  this  paper  or  by  new  user-generated 
issues.  An  example  of  this  process  of  organization  is  illustrated  by  the 
progression  presented  in  Figure  1 .  This  progression  begins  with  the  initial 
user-generated  question  and  ends  with  the  empirically  supported  information 
provided  to  the  ISD  decision  maker  in  answer  to  the  question.  In  this  example, 
the  user  is  interested  in  knowing  how  much  fidelity  is  needed  in  a  training 
device  used  in  training  a  specific  task.  This  question  is  then  framed  as  a 
basic  research  issue  to  be  addressed  by  the  research  and  development  (R6D) 
community.  This  overall  research  issue  concerns  the  effects  of  the  interaction 
of  level  and  type  of  fidelity  and  task  type  on  transfer  of  training.  This 
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TRAINING  DESIGN  GUIDANCE  SYSTEM 
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Figure  1.  A  simplified  training  device  research  issue  progression. 


issue  encompasses  many  research  areas  that  will  be  discussed  later.  To  pro¬ 
vide  a  comprehensive  empirically  based  answer  to  the  question,  an  adequate 
task  taxonomy  would  have  to  be  developed  and  applied  (see  Fleishman,  1982,  for 
review),  adequate  empirical  research  into  all  of  the  various  task-type  and 
fidelity  level  combinations  would  have  to  be  performed,  and  many  of  the  other 
interacting  variables  would  have  to  be  checked.  However,  it  does  not  seem 
reasonable  to  keep  a  training  program  developer  waiting  while  a  large  data 
matrix  is  developed  if  our  goal  is  only  to  use  a  small  part  of  the  matrix  to 
answer  the  original  question.  A  much  more  practical  approach  would  be  to 
identify  the  data  necessary  to  answer  the  user's  question  and  then  check  the 
previous  research  or  perform  new  research  in  order  to  answer  that  particular 
question.  Once  the  specific  question  is  answered,  the  data  base  can  serve  as 
an  archive  to  insure  that  future  researchers  will  not  have  to  duplicate 
efforts  when  a  similar  question  is  asked. 

To  some  extent,  the  process  of  centering  on  the  specific  situation  is  the 
typical  method  used  to  provide  information  to  the  training  program  decision 
maker.  An  organized,  accessible,  and  usable  knowledge  system  can  be  built  from 
the  data  existing  in  the  reports  generated  in  response  to  previous  trainer’s 
questions.  Such  a  system  would  incorporate  the  data  already  accumulated  in 
previous  efforts.  For  example,  the  Air  Force  is  in  the  process  of  developing 
a  data  base  called  the  Integrated  Perceptual  Information  for  Designers  (Boff 
&  Martin,  1980).  This  effort  centers  on  the  perceptual  information  requirements 
for  simulator  displays  and  could  be  a  major  component  of  an  overall  ISD  guid¬ 
ance  data  base.  Another  example  is  the  Design  of  Training  Systems  (DOTS) 
project  (Bellamy  et  al.,  1974;  Branch  et  al.,  1976;  Duffy  et  al.,  1977;  Miller 
<■  Duffy,  1975,  1976),  which  uses  computer  based  mathematical  models  as  tools 
to  manage  training  organizations.  The  proposed  system  would  serve  ISD  decision 
makers  by  making  all  of  the  existing  ISD  guidance  information  easily  available 
as  well  as  providing  a  framework  for  the  generation  of  new  data.  Hie  organiza¬ 
tion  of  the  data  base  and  knowledge  system  would  identify  the  specific  instances 


that  have  already  been  adequately  investigated  as  well  as  identifying  any 
areas  that  need  further  research.  The  primary  user  of  the  system  would  be  the 
ISD  decision  maker,  who  would  thus  be  able  to  use  all  of  the  empirically  based 
information  concerning  the  specific  situation. 

An  additional  benefit  to  both  users  and  researchers  would  be  that  de¬ 
cisions  made  by  training  developers  in  those  areas  that  have  not  been  em¬ 
pirically  investigated  could  also  be  included  in  the  data  base.  Many 
design  decisions  are  due  to  training  device  or  actual  equipment  changes, 
new  methods  of  training,  or  even  instructor  and  trainee  changes  and  require 
immediate  action  by  the  training  developer.  It  is  possible  that  these 
decisions  and  the  information  that  supported  the  decisions  could  be  entered 
into  the  data  base  in  order  to  develop  some  form  of  “consensus"  guidance  for 
that  situation.  This  approach  would  provide  immediate  SME  guidance  when 
empirical  data  are  not  available  and  would  alert  the  researcher  to  areas 
requiring  future  empirical  investigation.  As  a  final  note,  this  data  base 
organization  and  utilization  process  could  also  allow  identification  of 
possibly  less  important  situations,  preventing  unnecessary  experimentation. 

The  various  research  issues  which  we  will  discuss  do,  to  a  large  ex¬ 
tent,  reflect  questions  addressed  by  training  developers  almost  every  day. 

Each  of  these  questions  represents  a  specific  instance  in  a  progression  like 
that  in  Figure  1 .  Once  existing  data  have  been  accumulated  and  organized 
into  an  accessible,  user-oriented  knowledge  system,  the  training  developer's 
questions  can  be  more  easily  addressed  and  answered.  We  realize  that  many 
of  the  practical  questions  must  be  asked  and  empirical  data  from  previous 
research  evaluated  or  new  empirical  data  generated  as  the  initial  data  base 
and  knowledge  system  is  created.  The  overall  process  is  planned  as  an 
iterative  process,  as  illustrated  in  Figure  1 .  As  new  data  are  accumulated 
on  training  questions,  based  on  empirical  research  or  on  SME  guidance,  the 
data  would  be  included  as  updated  information  in  the  system.  A  means  of 
regularly  reviewing  and  updating  the  data  base  and  knowledge  system  would  be 
established  at  its  initial  creation. 


Accessing  the  Data  Base 

Just  as  important  as  the  creation  of  a  user-oriented  training  issues 
guidance  system  base  is  the  necessity  that  the  user  be  able  to  access  it, 
that  the  system  be  eminently  usable.  Two  candidate  methods  for  ensuring  that 
the  system  meets  this  criterion  are  a  guidebook  or  a  computer  based  system. 

Each  of  these  approaches  has  pluses  and  minuses  connected  with  its  development 
and  use.  Guidebooks  have  commonly  been  used  in  various  information  utilization 
efforts  prior  to  this,  but  only  recently  have  efforts  been  made  to  automate 
these  kinds  of  processes  (e.g.,  the  DSTIS  project  at  the  Army  Training  Support 
Center) . 

Guidebooks  would  certainly  be  the  easier  method  to  develop  in  the  initial 
stages  of  the  project.  The  information  organization  accumulated  from  existing 
research  would  be  easily  compiled  and  the  early  cross  indexing  of  the  guidebook 
would  be  based  entirely  on  ISD  decision  steps.  A  guidebook  may  be  readily 
accepted  by  the  general  training  community,  having  been  exposed  to  various 
types  of  guidebooks  previously.  There  would  be  only  minimal  training  involved 


in  using  a  guidebook  as  there  is  a  limit  to  the  amount  of  specific  indexing 
that  can  be  used  in  a  paper  format.  This  training  would  most  easily  be  pre¬ 
sented  in  the  guidebook  itself.  One  area  which  can  both  hurt  and  help  the 
effectiveness  of  the  guidebook  approach  is  that  guidebooks  typically  require 
non-native  users  (one  requires  some  degree  of  subject  matter  expertise)  who 
can  accurately  interpret  and  integrate  the  information  presented.  In  addition, 
as  the  guidebook  development  is  completed  and  widespread  use  begins,  guide¬ 
books  become  more  and  more  difficult  to  iterate  and  update.  Typically, 
unless  a  formal  mechanism  for  updates  is  established,  the  guidebook  ends  as 
a  single  edition  publication,  and  if  an  update  mechanism  is  used,  some  por¬ 
tion  of  the  trainer's  time  is  wasted  in  updating  the  book.  Perhaps  it  is 
because  of  these  problems  and  the  previously  mentioned  interpretation  diffi¬ 
culties  that  most  guidebooks  are  rarely  used  in  any  rigorous  fashion  (Heeringa, 
Baum,  Holman,  &  Peito,  1982)  For  these  reasons,  in  spite  of  the  initial  ease 
of  guidebook  organization,  we  believe  that  a  computer  based  approach  would  be 
much  more  useful  in  the  long  term. 

First  of  all,  the  Army  is  beginning  to  use  new  computer  based  systems  to 
control  the  data  and  paperwork  flow  required  for  the  development  of  training 
devices  and  training  systems.  An  example  is  the  Devices  and  Systems  Training 
Information  System  (DSTIS)  implemented  at  Fort  Eustis  by  the  Army  Training 
Support  Center  (ATSC)  for  TRADOC,  DARCOM,  and  training  school  use.  There  is 
also  a  system  under  development  for  aiding  training  development  during  the 
concept  formulation  stage  in  the  acquisition  of  a  major  weapon  system.  It  is 
called  the  Early  Training  Estimation  System  (ETES)  and  is  in  the  early  stages 
of  development  at  Fort  Bliss.  It  will  in  part  be  based  on  comparing  preliminary 
concepts  with  currently  existing  training  devices  and  programs.  These  computer 
based  systems  are  immature  in  that  they  are  not  yet  fully  implemented,  although 
there  are  plans  to  link  these  systems  together  for  information  transmission 
and  data  sharing.  Typically  these  systems  are  projected  to  solve  certain 
specific  problems  that  the  different  commands  face,  although  the  overlap  of 
actual  end-users  is  considerable.  These  systems,  when  integrated,  would  begin 
to  solve  the  common  Army-wide  problems  mentioned  above,  of  tracking  and  updat¬ 
ing  the  documentation  and  of  including  new  guidance  information  while  allowing 
easier  end-user  access.  However,  merely  integrating  the  above  systems  will 
not  solve  the  problem  of  utilizing  empirical  data  and  deriving  guidance  from 
that  data.  How  to  translate  empirical  data  into  user-oriented  design  guidance 
is  not  addressed  by  the  types  of  developing  systems  mentioned  above. 

An  automated  training  design  guidance  system  presents  a  pattern  that  is 
opposite  to  the  advantages  and  disadvantages  of  a  guidebook.  Early  in  the 
development  of  the  system  there  will  be  problems  of  hardware  and  software 
selection.  In  compiling  the  data  base  the  work  required  to  adequately  organize 
the  existing  empirical  information  will  also  be  more  extensive  than  that  re¬ 
quired  by  a  simple  guidebook.  Constructing  the  rule  based  system  for  indexing 
the  data  base  and  organizing  the  guidance  system  (including  changing  the  system 
in  response  to  SME  provided  information)  will  certainly  be  more  difficult  than 
indexing  a  guidebook  but  will  alleviate  the  problem  of  expert  interpretation 
of  research  results  by  providing  direct  recommendations  to  the  user.  The  rule 
system  will  thus  aid  and  be  improved  by  the  SMEs  while  providing  assistance  to 
more  naive  users  and  facilitating  the  organization  of  a  large  body  of  knowledge. 
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The  first  step  in  the  organization  of  an  empirical  data  base  is  to  identi¬ 
fy  the  domain  of  information  to  be  used.  An  organizing  framework  (see  struc¬ 
turing  section  below)  is  used  in  this  paper  which  supports  the  linking  of  many 
variables  in  the  training  device  research  domain.  The  second  step  is  to  review 
and  reorganize  existing  literature  for  inclusion  in  the  data  base,  this  paper 
begins  these  two  steps  by  providing  a  conceptual  organization  for  the  data 
base  and  initially  reviewing  the  literature.  The  goal  is  a  data  base  that  can 
be  manipulated  in  several  ways:  via  standard  keyword  search,  searches  on  train¬ 
ing  device  type  or  other  training  variables,  and  by  meta  analysis  programs 
(under  development  by  the  Army).  Such  data  base  manipulations  will  facilitate 
the  delivery  of  design  guidance  based  on  empirical  data  on  the  training  effects 
of  different  training  system  variables. 

There  is  a  considerable  body  of  commonly  held  knowledge  about  training,  as 
evidenced  in  Army  regulations,  technical  reports,  and  guidebooks.  There  is  also 
considerable  knowledge,  for  example  about  fidelity  requirements,  that  is  buried 
in  the  literature  and  that  can  be  extracted  by  meta  analytic  or  delphic  review. 
Existing  knowledge  would  thus  form  the  basis  of  a  computerized  guidance  system 
for  the  early  specification  of  training  device  requirements  and  would  provide 
empirically  based  input  to  the  immature  Army  systems  mentioned  above. 

New  computer  techniques  based  on  artificial  intelligence  research  are 
providing  better  ways  of  dealing  with  knowledge  (as  the  distillation  of  data). 
These  expert  systems  are  essentially  computerized  subject  matter  experts  that 
operate  in  very  restricted  knowledge  domains.  The  approach  advocated  in  this 
paper  differs  from  existing  expert  systems  in  that  the  proposed  expert  system 
will  be  based  on  knowledge  drawn  from  an  empirical  data  base  of  structured 
research,  rather  than  on  the  knowledge  and  rules  drawn  from  a  group  of  acknowl¬ 
edged  masters  in  the  training  field.  This  strategy  is  proposed  for  two  main 
reasons:  a  group  of  acknowledged  masters  does  not  exist  and  a  data  base  allows 
us  to  iterate  and  improve  our  knowledge  as  new  data  are  developed.  We  can 
thus  continually  improve  the  guidance  available  from  the  system.  The  proposed 
knowledge  system  would  draw  on  the  empirical  data  base  to  provide  reasoned 
conclusions  from  various  analyses  of  the  data.  Such  a  system  would  furnish 
both  direct  guidance  to  ^training  device  and  instructional  system  designers  and 
also  provide  empirical  and  analytical  support  to  training  device  and  instruc¬ 
tional  system  researches.  The  advantage  of  an  iterative  system  is  that  short 
term  guidance,  based  on  existing  knowledge  may  be  provided  fairly  quickly. 

This  guidance  may  be  constantly  improved  as  new  data  are  developed  frost  research 
and  are  incorporated  into  the  system.  We  can  thus  have  the  best  of  both  worlds: 
immediate  guidance  and  continually  improving  guidance.  The  concept  development 
for  the  data  base  and  expert  system  is  currently  being  studied  at  ARI . 

Once  these  initial  developmental  problems  have  been  dealt  with,  the  com¬ 
puter  system  provides  many  unique  and  beneficial  capabilities  to  the  training 
community.  The  system  will  allow  almost  immediate  updating  based  on  new  re¬ 
search  results  while  alerting  Any  researchers  to  the  most  important  problems 
faced  by  trainers  in  the  field.  In  many  ways  the  automated  system  would  enable 
the  trainer  to  become  more  of  a  researcher,  being  in  direct  and  immediate 
control  of  continuing  evaluations  of  their  own  programs.  To  some  degree 
the  amount  of  background  work  would  be  lessened  for  the  trainer,  in  that  a 
networked  system  would  decrease  the  amount  of  manual  updating  required,  as 
mentioned  above,  while  possibly  enhancing  communications  within  the  overall 


7 


Army  training  community.  Finally,  and  perhaps  most  importantly  in  terms  of 
future  efficiency,  computer  equipment  in  the  trainer's  office  should  increase 
both  quality  and  productivity  while  simultaneously  easing  the  workload  problems 
involved  in  maintaining  and  increasing  Army  training  effectiveness. 


Structuring  the  Data  Base 

In  order  to  establish  a  data  base  that  is  both  useful  as  the  foundation  for 
a  rule  based  expert  system  and  that  is  responsive  to  Army  user  needs,  it  is  cru¬ 
cial  that  an  appropriate  conceptual  structure  for  the  data  base  be  established. 
This  conceptual  structure  will  not  only  help  accumulate  existing  data,  but  will 
also  sensitize  researchers  to  areas  requiring  new  data. 

As  discussed  above  (see  Figure  1),  it  is  user  questions  which  stimulate 
researchers  to  conduct  their  investigations.  In  the  following  sections,  an 
Instructional  Systems  Design  (ISD)  framework  is  used  to  organize  research 
questions.  Such  research  questions  require  empirical  data  if  they  are  to 
support  the  proposed  data  base  and  expert  system.  The  accumulation  of  these 
research  questions  is  based  on  an  extensive  literature  review  and  the  questions 
will  serve  as  the  basic  conceptual  structure  for  the  training  system  data  base 
and  the  knowledge  system  to  be  derived  from  the  empirical  data  base. 


RESEARCH  ISSUES  IN  ISD  CONTEXT 

Instructional  Systems  Development  (ISD)  encompasses  many  diverse  activi¬ 
ties  in  addition  to  training  device  design  (TRADOC,  1975).  Tasks  must  be 
analyzed,  training  needs  established,  instructional  media  selected,  programs 
of  instruction  established,  and  training  effectiveness  assessed.  From  a  system 
perspective,  all  of  these  activities  are  equally  as  important  as  the  design 
of  training  device  characteristics.  A  change  in  output  from  any  ISD  stage 
will  affect  all  others  since  they  fora  an  interdependent  system.  Likewise, 
the  research  issues  which  provide  data  for  training  device  design  may  also 
provide  necessary  data  for  other  ISD  activities.  When  determining  how  research 
should  be  conducted,  it  is  therefore  important  to  recognize  the  many  ways  in 
which  these  data  may  be  used,  as  well  as  their  important  derivative  effects 
through  the  entire  ISD  process. 

The  interservice  ISD  procedures  (TRADOC,  1975)  depicts  a  5-stage  process. 
Other  descriptions  of  ISD  have  used  up  to  20  steps  (Nechner,  1981).  The  char¬ 
acteristic  shared  by  all  descriptions  of  ISD  is  the  interdependency  of  its 
stages.  Th<  terdependency  of  the  ISD  process  is  illustrated  in  Figure  2, 
which  pre  d  very  simplified  overview  of  ISD.  This  simplified  diagram 
depicts  iS  ontinuous  process  with  four  major  sections  (shown  in  bold¬ 
face  typtO-  stage  view  of  ISD  is  focused  on  training  device  design. 

It  is  not  p*  sf  as  the  best  or  the  only  version  of  ISD,  but  rather  is 
presented  ie>ns  of  organizing  the  research  issues  to  be  discussed  below. 

Other  approaches  to  ISD  may  have  less  emphasis  on  training  devices;  however, 
this  paper  is  organized  around  training  device  design.  Training  developers 
concerned  with  other  aspects  of  ISD  must  still  deal  with  most  of  the  research 
issues  discussed  below  due  to  the  systemic  nature  of  training  programs. 

Although  iterative,  the  ISD  process  may  be  thought  of  as  beginning  with  an 


assessment  of  training  needs.  Training  needs  are  recognized  when  current 
training  is  determined  to  be  inadequate  either  by  assessing  external  threats 
(e.g.,  a  new  weapon  system)  or  by  making  internal  assessments  of  current 
training  effectiveness.  In  any  situation  where  current  training  fails  to 
reach  a  desired  level  of  effectiveness,  a  training  need  exists. 


External  Threat  Assessment  -  Technology 

Readiness  Assessment  v.  —  _  Development 


Figure  2.  A  simplified  overview  of  the  Instructional  Systems 
Development  process. 


Training  needs  are  typically  met  by  changing  the  conventional  instruc¬ 
tional  delivery  system.  Several  methods  may  be  used  to  change  an  instructional 
delivery  system  in  response  to  established  needs.  One  method  is  to  adapt  or 
alter  the  current  program  of  instruction  (POI).  Another  approach  would  develop 
a  new  training  device  and  adapt  the  existing  POI  to  the  new  device.  Finally, 
both  a  new  POI  and  a  new  training  device  might  be  developed  concurrently.  Any 
of  these  strategies  leads  to  the  determination  of  new  training  needs,  and  at 
this  point  the  cycle  begins  again. 

The  ISD  process  is  a  dynamic,  iterative  one,  and  there  are  many  research 
issues  which  may  provide  data  for  the  necessary  judgments  at  various  points  in 
the  ISD  process.  The  research  issues  discussed  below  are,  in  essence,  struc¬ 
tured  ways  of  observing  and  recording  the  activities  that  connect  the  four 
sections  depicted  in  Figure  2.  A  key  point  is  that  any  research  issue  should 
be  examined  as  it  relates  to  each  portion  of  the  ISD  process.  In  the  following 


section,  specific  research  issues  are  defined  and  a  preliminary  strategy  for 
examining  how  each  issue  might  influence  training  device  design  as  well  as  the 
the  other  interrelated  aspects  of  the  ISD  process  is  discussed. 


DEFINITION  AND  ELABORATION  OF  RESEARCH  ISSUES  IN  TRAINING  DEVICE  DESIGN 

As  stated  above,  no  research  issue  exists  in  isolation  if  viewed  from  an 
ISD  perspective.  While  this  paper  centers  on  training  device  design,  it  is 
impossible  to  discuss  design  without  also  discussing  other  aspects  of  the  ISD 
process.  Table  1  presents  a  set  of  training  system  research  issues  derived 
from  the  training  literature  which  are  believed  to  be  major  influences  in  the 
effectiveness  of  training.  Each  of  the  research  issues  identified  in  Table  1 
will  be  discussed  from  two  perspectives,  first  within  the  context  of  the  major 
section  of  the  ISD  process  with  which  it  is  most  concerned  and  second  from  the 
perspective  of  its  implications  for  training  device  design.  It  should  again 
be  emphasized  that  every  issue  interacts  with  all  others  and  eventually  it  is 
the  effects  of  alj.  these  interactions  which  must  be  determined. 


Training  Needs  Assessment  Issues 

The  first  set  of  issues  is  concerned  with  assessing  training  needs.  Be¬ 
fore  any  modification  of  an  existing  training  system  or  the  creation  of  a  new 
training  system  can  begin,  it  must  first  be  established  that  there  is,  in  fact 
a  training  deficit.  A  task  analysis  is  an  important  step  in  determining  the 
existence  of  such  deficits. 

Task  Analysis.  There  are  various  approaches  to  the  analysis  of  tasks 
(Hays,  1981a),  and  each  approach  produces  a  different  output  depending  upon  how 
one  wishes  to  use  the  analysis  (see  Appendix  A  for  more  detail).  A  task  anal¬ 
ysis  which  is  to  be  used  to  develop  proficiency  tests  for  a  training  course 
will  generate  a  different  output  from  a  task  analysis  intended  to  aid  in  train¬ 
ing  device  design.  For  example,  the  developer  of  proficiency  tests  requires 
task  analysis  information  on  the  indicators  of  successful  performance  (e.g., 
number  of  targets  hit,  time  to  repair  a  component,  critical  malfunctions, 
etc.).  On  the  other  hand,  the  training  device  designer  is  concerned  with 
transfer  of  training  from  the  training  device  to  the  actual  equipment  and 
therefore  requires  task  analysis  information  about  the  controls,  displays, 
and  information  required  to  complete  the  task.  The  designer  then  must  decide 
how  best  to  represent  aspects  of  the  actual  equipment  in  the  training  device. 

A  task  analysis  used  in  media  selection  will  have  still  a  third  type  of  output. 
The  training  medium  selected  will  depend  on  an  assessment  of  training  techniques 
and  task  criticality  (Reiser  et  al.,  1981)  and  therefore  requires  a  task  anal¬ 
ysis  output  which  addresses  these  requirements.  If  each  user  conducts  a  separate 
task  analysis,  much  of  the  information  in  these  analyses  may  be  duplicated, 
although  certain  information  that  is  needed  for  one  kind  of  decision  (e.g., 
media  selection)  will  not  be  necessary  for  other  decisions  (e.g.,  proficiency 
test  design)  and  therefore  might  be  overlooked  in  a  task  analysis  with  a  goal 
that  is  too  specific. 


Table  1 


Training  System  Research  Issues 


ISD  Section  Research  Issues 


I  Training  Needs  Assessment  a.  Task  Analysis 

b.  Media  Selection 

c.  Trainee  Characteristics 

d.  Training  Strategy  Issues 

(1)  education  vs.  training 

(2)  generic  vs.  specific  training 

(3)  whole  vs.  part  task 

(4)  error  free  vs.  trial  t  error  learning 

(5)  self-paced  vs.  lock  step  training 

II  Training  Device  Design  a.  Physical-Functional  Fidelity  Interaction 

b.  Augmented  Feedback  &  Instructional 
Features  Issues 

c.  Effects  of  Specific  Architectural 
Features 

(1)  2D  vs.  3D 

(2)  trainee  input  devices  (touch  panel 
vs.  key  pad,  etc.) 

(3)  student  station/instructor  station 
ratio 

III  Training  Device  Implementation  a.  User  Acceptance  Issues 

( 1 )  instructors 

(2)  trainees 

b.  Integration  of  Training  Device  into 
Program  of  Instruction 

c.  Non institutional  Use  of  Training 
Devices  and  Simulators 

a.  General  Training  Evaluation  Measurement 

Issues 

( 1 )  process  vs .  outcome  measures 

(2)  objective  vs.  subjective  measures 

(3)  criterion  vs.  norm  referenced  measures 

(4)  internal  and  external  validity 

b.  Types  of  Effectiveness  Assessment 

(1)  establishment  of  proficiency  standards 

(2)  transfer  of  training  vs.  other 
measures 

(3)  cost  effectiveness  vs.  training 
effectiveness 

(4)  short-term  vs.  long-term  retention 
studies 

d.  Resource  Requirements 


IV  Training  Effectiveness 
Assessment 


Research  is  necessary  to  determine  just  how  task  analyses  are  currently 
conducted  and  how  their  outputs  are  used.  At  this  time  there  is  no  truly 
standardized  task  analysis  methodology  (Fleishman,  1962).  Many  task  analyses 
are  performed  at  the  last  minute  to  fulfill  a  procedural  requirement  (TRADOC 
PAM  350-30,  1975)  or  are  shortcut  to  save  resources  (Branson,  1981;  Kane  & 
Holman,  1982).  Surveys  of  training  device  procurements  could  detail  who  per¬ 
formed  the  task  analysis,  how  the  task  analysis  was  accomplished,  what  types 
of  information  were  generated,  and  how  this  information  was  used.  A  compendium 
of  these  survey  data  could  then  aid  in  specifying  a  standardized  task  analysis 
format  and  procedure.  ARI  is  currently  beginning  this  effort  by  surveying  the 
front  end  analysis  procedures  used  in  the  design  of  the  bread  board  devices  in 
the  Army  Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS)  pro¬ 
gram.  A  standardized  task  analysis  format  which  includes  all  information  neces¬ 
sary  in  all  ISD  phases  would  greatly  reduce  duplication  of  effort  since  re¬ 
peated  task  analyses  for  various  parochial  decisions  would  not  have  to  be 
conducted.  Hays  (1981a)  suggests  how  the  existing  TRADOC  task  analysis  format 
might  be  restructured  to  provide  all  this  necessary  information.  The  use  of 
such  a  task  analysis  format  by  someone  trained  in  the  procedures  required  to 
conduct  the  analysis  would  provide  savings  of  time  and  money  for  anyone  who 
requires  task  analysis  information.  This  is  especially  true  in  the  case  of 
training  device  design  because  decisions  about  the  characteristics  of  the 
device  must  be  driven  in  good  part  by  information  on  what  must  be  trained 
(Hays,  1981a). 

Research  is  also  needed  to  develop  a  workable  task  taxonomy  (Fleishman, 
1982).  Classifying  tasks  into  a  manageable  number  of  categories  would  greatly 
facilitate  empirical  investigations  of  how  different  tasks  interact  with 
training  devices  of  different  designs.  Such  a  taxonomy  would  also  aid  in  the 
process  of  developing  generalized  design  guidance  which  could  refer  to  task 
types  rather  than  specific  tasks.  This  research  would  be  (1)  analytic, 
surveying  existing  taxonomies  and  previous  empirical  research,  (2)  creative, 
developing  improved  taxonomies,  and  (3)  empirical,  validating  any  candidate 
taxonomies  and  generating  empirical  data  on  the  interaction  of  task  type  with 
training  device  characteristics. 

Media  Selection.  Even  before  decisions  are  made  about  what  characteristics 
a  training  device  should  exhibit,  the  decision  must  be  made  to  use  a  training 
device  rather  than  some  other  medium  of  instructional  delivery.  There  are  at 
least  20  different  instructional  media  types,  ranging  from  actual  equipment 
trainers  (AETs)  and  simulators  to  films  and  classroom  instructors,  that  are 
candidates  for  selection  as  part  of  a  training  delivery  system  (Reiser  et 
al.,  1981).  This  selection  should  be  based  on  training  needs  (Caro,  1977a; 
Smode,  1971)  and  should  be  firmly  embedded  in  the  ISD  process.  Research  is 
necessary  to  validate  and  possibly  modify  existing  media  selection  models 
(such  as  Reiser  et  al.,  1981;  Anderson,  1976;  Branson  et  al.,  1975;  Romiszowski, 
1974;  Braby,  1973;  Bretz,  1971;  Boucher  et  al.,  1971;  Campbell  &  Hughes,  1978) 
and  also  to  extend  the  media  selection  models  to  include  specifications  for 
the  characteristics  of  whatever  medium  is  selected  rather  than  stopping  with 
the  selection  of  a  given  medium.  This  is  even  more  important  for  training 
devices  and  simulators,  which  are  much  more  expensive  and  difficult  to  design, 
than  for  simpler  instructional  media  such  as  slide  projectors.  Empirically 
derived  data  on  the  current  uses  of  media  selection  models  can  tell  us  whether 
a  consistently  applied  model  could  improve  Army-wide  training. 


It  may  prove  necessary  to  employ  several  types  of  instructional  media  at 
various  stages  during  a  program  of  instruction  (Kinkade  and  Wheaton,  1972). 

If  this  is  the  case,  then  research  will  also  be  necessary  to  determine  the 
training  effectiveness  of  the  various  media  mixes.  Empirically  derived  data 
are  required  to  determine  the  most  effective  use  of  instructional  media  at 
each  stage  of  training  and  also  to  determine  how  the  transition  from  one  medium 
to  another  can  be  accomplished  with  the  most  effective  training  outcome.  As 
will  be  discussed  later,  research  is  also  needed  to  determine  the  effect  of 
the  interaction  of  media  types  and  media  mixes  with  trainee  characteristics 
and  training  strategies.  The  AMTESS  program  is  providing  initial  data  on  the 
mix  of  3D  mockups  and  2D  instructional  delivery  systems  (e.g.,  CRTs  or  slide 
projectors) . 

Trainee  Characteristics.  Not  every  trainee  comes  to  training  with  the 
same  competencies,  experience,  confidence,  or  motivational  structures.  Depend¬ 
ing  on  the  task  type,  training  strategy,  and  training  goal,  each  individual's 
cognitive  characteristics  may  influence  the  success  of  the  instructional  pro¬ 
gram  for  that  person.  Research  could  help  to  specify  the  appropriate  train¬ 
ing  device  characteristics  and  training  strategies  for  different  types  of 
trainees.  For  example,  a  novice  trainee  may  not  have  the  experience  necessary 
to  benefit  from  all  the  cues  available  with  a  high-fidelity  training  device. 

On  the  other  hand,  an  expert  trainee  may  require  a  high-fidelity  device  to 
provide  all  the  necessary  information  for  learning  the  task  (Martin  and  Waag, 
1978a). 

Research  is  also  needed  to  determine  what  enabling  skills  are  required 
for  each  type  of  training  device  (e.g.,  reading  ability  to  communicate  with  a 
computer)  and  also  to  determine  whether  trainees  should  be  selected  for  a  train¬ 
ing  device  or  a  training  device  designed  for  the  trainees.  If  it  is  determined 
that  trainee  selection  is  the  appropriate  strategy,  then  further  research  will 
be  required  to  validate  current  selection  procedures.  Many  of  these  selection 
procedures  use  generic  paper-and-pencil  tests  to  assess  trainee  abilities.  Hie 
problem  is  that  the  generic  paper-and-pencil  tests  used  to  determine  trainee 
abilities  may  have  low  correlation  with  actual  skill  performance  (Foley,  1977). 

Motivation  is  a  characteristic  that  may  have  a  strong  influence  on  learn¬ 
ing,  retention,  and  performance.  Schneider  (1982)  has  advanced  soeie  specific 
guidelines  for  rapid  training  of  subtasks  to  mastery  levels  and  points  out 
that  a  major  variable  for  the  success  of  the  training  process  is  the  motiva¬ 
tional  level  of  the  trainee.  Research  into  trainee  confidence  has  shown  that 
as  much  as  58%  of  the  difference  between  attainment  scores  and  measured  intel¬ 
ligence  may  be  explained  by  the  manipulation  of  this  factor  (Sylvester,  1970). 
Clearly  there  is  a  need  for  clarification  of  these  motivational  issues  as  they 
relate  to  military  training.  For  example,  how  can  a  trainee's  motivation  be 
improved,  how  will  improved  motivation  affect  training  and  how  is  motivation 
affected  by  specific  training  device  designs. 

Another  question  that  may  influence  training  device  design  and  selec¬ 
tion  is  the  effect  of  the  trainee's  learning  set  on  training  outcome  (Monge, 
1969).  Hie  notion  of  learning  sets  or  learning  to  learn  (Harlow,  1949»  Harlow, 
1959)  suggests  that  learning  a  set  of  similar  problems  can  result  in  the  acquisi¬ 
tion  of  a  generalized  skill  that  can  later  be  successfully  applied  to  new  and 
different  problems  of  the  same  class.  Harlow's  work  was  done  with  nonhuman 
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subjects  and  subsequent  studies  have  used  human  children  as  subjects  (Rydberg 
&  Amberg,  1976;  Fagen,  1977).  To  the  authors'  knowledge,  there  have  been  no 
studies  which  investigated  the  effects  of  learning  set  in  a  military  context. 

How  the  trainee  approaches  the  training  situation  may  hinder  or  help  the  learn¬ 
ing  process.  The  trainee's  knowledge,  how  that  knowledge  is  organized  and  ‘ 

used,  as  well  as  the  trainee's  cognitive  skills  may  have  important  effects 
on  the  success  of  the  training  program  (Ausburn,  Ausburn  &  Ragan,  1980).  Re¬ 
search  is  required  to  define  learning  sets  more  clearly  as  well  as  to  determine  ' 

their  effects  on  training.  t 

S 

There  is  some  evidence  to  indicate  that  neurological  organization  in  terms 
of  hemispherical  dominance  is  unimportant  for  procedural  motor  skills  (Tyler, 

1971).  There  may,  however,  be  neurological  differences  that  affect  other  types 
of  learning  and  that  would  therefore  influence  acquisition  of  some  of  the  more 
complex  troubleshooting  skills  (Ausburn,  Ausburn  &  Ragan,  1980).  Empirical 
research  in  this  area  could  clarify  the  relationship  between  neurological 
organization  and  specific  training  and  task  demands. 

The  need  for  mastery  of  complex  troubleshooting  skills  has  been  identified 
by  surveys  indicating  that  there  is  a  high  degree  of  human  error  in  maintenance 
tasks  (Robinson  et  al.,  1970;  Orlansky  &  String,  1981b).  As  military  systems 
become  more  complex,  the  cognitive  demands  on  the  trainee  will  also  increase, 
as  will  the  cost  of  errors.  This  increasing  complexity  in  the  military  en¬ 
vironment  requires  that  many  aspects  of  trainee  characteristics  and  task  re¬ 
quirements  be  studied  in  order  for  training  to  produce  the  best-trained 
persons  for  the  job.  Research  questions  should  center  on  how  the  demands  of 
various  training  device  designs,  interacting  with  trainee  characteristics, 
affect  task  learning,  transfer  of  training,  as  well  as  long-term  retention  and 
performance. 

Training  Strategy  Issues.  The  recurrent  theme  throughout  this  paper  is 
the  interaction  of  all  training  system  research  issues.  This  point  is  made 
again  in  the  discussion  of  training  strategy  issues.  Once  the  task  analysis 
is  performed,  the  integrated  process  of  training  strategy  development  and 
media  selection  based  on  trainee  characteristics  and  training  needs  can  begin. 

The  first  general  question  which  must  be  answered  in  developing  an  overall 
training  strategy  is:  Should  the  focus  be  on  education,  centering  on  teaching 
general  principles,  or  should  the  focus  be  on  training  that  provides  only 
task-specific  information?  If  the  performance  of  the  target  task  requires 
psychomotor  skills,  the  program  of  Instruction  should  include  spatial  and 
kinesthetic  training  that  at  least  approximates  the  psychomotor  skills  needed 
for  criterion  performance.  However,  if  the  target  task  or  set  of  tasks  covers 
variable  situations  that  require  general  problem-solving  skills,  to  some  degree 
the  training  program  should  be  a  general,  discovery-oriented  curriculum  that 
enhances  understanding  and  strategy  acquisition  (Singer,  1977).  In  many  main¬ 
tenance  tasks  a  major  portion  of  the  training  requirements  involves  the  correct 
use  and  basic  understanding  of  technical  manuals,  which  may  be  best  taught  by 
general  instruction  and  practice  with  common  information-handling  tasks  (Hogan, 

1978).  If  entry-level  people  are  lacking  in  necessary  skills,  then  perhaps  a 
general  educational  program  oriented  toward  generic  tasks  should  precede  train¬ 
ing  for  more  specific  tasks. 
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A  second  question  which  should  be  addressed  is  related  to  the  issue  dis¬ 
cussed  above:  Is  it  more  effective  to  train  for  generic  tasks  (e.g.,  tasks 
applicable  to  all  types  of  automotive  engines)  or  is  it  more  effective  to 
train  for  specific  tasks  (e.g.,  tasks  applicable  only  to  a  single  engine  type)? 
The  underlying  psychological  issue  concerns  transfer  of  training.  Empirical 
research  can  determine  whether  skills  learned  on  generic  tasks  transfer  to 
specific  tasks  and  vice  versa.  If  there  is  a  high  degree  of  transfer,  then 
other  factors  (i.e.,  cost)  can  be  used  to  decide  on  what  training  strategy  to 
develop.  If  the  criterion  task  demands  on  the  trainee  cover  a  class  of  equip¬ 
ment,  or  transfer  from  the  specific  task  training  is  low,  then  it  may  be  more 
effective  and  efficient  to  use  a  broader-based  generic  training  approach.  It 
may  also  prove  effective  to  provide  an  instructional  progression  from  generic 
to  specific  tasks.  It  should  be  evident  that  a  thorough  task  analysis  and 
training  effectiveness  assessment  program  is  essential  to  aid  in  making  these 
training  strategy  decisions. 

Within  an  instructional  program  that  trains  multiple  tasks,  a  third  basic 
question  concerns  whether  to  train  whole  tasks  or  part  tasks.  Very  few  tasks 
are  simple  enough  to  be  taught  from  start  to  finish  without  confusing  and/or 
overloading  the  naive  trainee.  Therefore  many  tasks  should  perhaps  be  dis¬ 
mantled  into  coherent  subtasks  for  the  purposes  of  training.  However,  if  the 
organization  or  interrelation  of  the  subtasks  is  very  high  and  the  whole  task 
is  not  overly  complex,  then  perhaps  the  task  should  be  taught  as  a  whole  (Blum 
&  Naylor,  1968).  Decisions  about  how  best  to  teach  various  task  types  should 
be  determined  empirically. 

A  fourth  general  question  that  must  be  answered  in  the  development  of  a 
training  strategy  concerns  the  pacing  of  the  overall  instructional  program. 

The  two  basic  approaches  to  instructional  pacing  are  lock  step  (time  based) 
and  self  paced  (performance  based).  Some  research  suggests  that  a  standard¬ 
ized,  lock-step  training  program  is  inefficient  in  that  the  training  forces 
some  large  percentage  of  learners  to  be  overtrained  in  order  to  bring  a 
majority  (or  all)  of  the  learners  up  to  some  criterion  performance  (Lawrence, 
1954).  One  conclusion  from  this  research  is  that  some  form  of  self-paced  or 
mastery  training  may  be  more  desirable  (Bloom,  1974).  On  the  other  hand,  the 
resource  constraints,  in  terms  of  number  of  instructors  and  instructor  time, 
has  led  some  Army  schools  to  require  lock-step  instruction  (e.g.,  the  Army 
Air  Defense  School  was  forced  to  go  to  lock-step  because  of  lack  of  instruc¬ 
tors).  the  decision  to  go  lock  step  or  self  paced  must  be  based  on  several 
factors:  a)  cost — whether  one  form  of  instruction  is  more  cost  effective  both 
in  terms  of  monetary  and  personnel  costs;  b)  performance — whether  all  students 
must  reach  the  same  level  of  proficiency  or  whether  some  range  of  skills  is 
adequate;  c)  transfer — whether  the  transfer  to  the  criterion  task  is  facilitated 
by  one  form  of  instruction  over  the  other.  Empirical  data  are  required  in 
order  for  the  Army  to  make  truly  informed  decisions  on  these  issues. 

One  additional  question  that  must  be  answered  when  designing  a  training 
strategy  is  whether  during  training  we  allow  the  trainee  to  make  mistakes 
(trial-and-error  learning)  or  whether  we  mitigate  his  probability  of  making 
mistakes  (error-free  learning).  Training  devices  can  be  designed  to  guide 
and  control  every  response  made  by  the  learner  or  they  can  allow  the  trainee 
to  make  mistakes  in  the  course  of  instruction.  If  the  target  task  is  normally 
performed  using  fixed  responses  and  has  built  in  prompts  and  cues,  then  a 
guided  learning  approach  in  which  trainees  are  not  allowed  to  make  many  errors 
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may  be  the  best  technique  (Singer,  1977).  On  the  other  hand,  if  problem  solv¬ 
ing  is  a  major  component  of  the  criterion  task,  then  there  are  indications  that 
a  high-error- rate,  discovery-based  technique  may  be  more  appropriate  (Singer, 
1977).  In  any  case,  there  is  a  considerable  base  of  research  on  this  issue, 
and  the  task  is  one  of  applying  what  is  known  to  the  integration  of  training 
device  design  with  training  strategy. 

Each  of  the  above  questions  must  be  answered  in  the  context  of  the  task 
to  be  trained  and  the  characteristics  of  the  trainees  as  well  as  the  character¬ 
istics  of  the  training  device  to  be  employed.  Research  is  necessary  to  estab¬ 
lish  the  strengths  and  weaknesses  of  each  training  approach  so  that  informed 
decisions  can  be  made  about  training  strategies. 

There  have  been  attempts  to  develop  learning  guidelines  and  algorithms 
for  training  various  types  of  tasks  (Aagard  &  Braby,  1976;  TRADOC,  1975). 
Research  is  needed  to  validate  these  guidelines  and  also  to  determine  how 
they  are  currently  being  used  and  how  they  might  be  used  more  effectively. 


Training  Device  Design  Issues 

As  previously  noted,  the  focus  of  this  paper  is  on  training  device  design 
issues.  In  each  section  of  this  paper,  issues  are  related  to  device  design, 
even  if  their  main  area  of  concern  is  another  aspect  of  ISD.  In  this  section, 
however,  we  discuss  research  issues  that  are  directly  relevant  to  training 
device  design.  Table  2  is  a  listing  of  theoretical  and  review  articles  which 
address  training  device  design  issues  by  examining  the  role  of  training  device 
fidelity  and  other  training  system  issues.  A  similar  listing  of  empirically 
based  studies  is  provided  in  Table  3.  It  should  be  remembered  that  the  factors 
which  are  examined  under  the  rubric  of  training  device  design  issues  will  be 
modified  by  the  interaction  with  variables  studied  under  the  rubric  of  other 
research  issues,  which  are  less  directly  relevant  to  training  device  design 
and  which  are  discussed  in  other  sections  of  the  paper. 

Training  Device  Fidelity  Issues.  The  term  fidelity  has  been  used  differently 
by  many  authors  (Hays,  1980).  For  our  purposes,  training  device  or  simulator 
fidelity  refers  to  the  degree  of  similarity  between  the  training  device  and 
the  actual  equipment.  Fidelity,  as  we  define  it,  has  two  components:  physical 
similarity  (how  the  device  looks)  and  functional  similarity  (how  the  device 
works).  Most  studies  of  the  effects  of  fidelity  have  compared  whole  devices 
rather  than  systematically  examining  the  interaction  of  these  two  aspects  of 
fidelity  (see  Table  2).  It  is  likely  that  physical  and  functional  fidelity 
are  complexly  related  since  change  in  one  will  to  some  degree  affect  the  other, 
but  not  necessarily  in  a  linear  fashion.  It  is  more  likely  that  the  two  aspects 
of  fidelity  are  analogous  to  the  two  aspects  of  color:  brightness  and  hue. 

When  either  aspect  of  color  is  changed,  one’s  perception  of  the  other  aspect 
is  also  changed  (Schiffman,  1976).  Systematic  research  is  needed  to  determine 
how  the  relative  values  of  each  aspect  of  fidelity  interact  with  other  training 
system  variables  to  produce  various  degrees  of  transfer  of  training.  Some  of 
the  training  system  variables  believed  to  interact  with  fidelity  are  task 
type,  task  difficulty,  required  skills,  trainee  sophistication,  stage  of 
training,  training  context,  integration  of  training  devices  into  programs  of 
instruction  (POIs),  user  acceptance,  and  use  of  specific  instructional  features 
(Hays,  1981a).  Current  efforts  (Baum  et  al.,  1982;  In  press)  are  laying  the 
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Table  2 


Representative  Training  Device  Design  and  Training  System  Theoretical 

and  Review  Articles 


Area  of  Flight  Simulation 


-  Adams,  J.  (1957) 

-  AGARD  (1980) 

-  Albery,  Gum  &  Kron  (1978) 

-  Bailey  &  Hughes  (1980) 

-  Caro  (1970,  1973b,  1977a,  1977b,  1979) 

-  Condon  et  al.  (1979) 

-  Cream,  Eggemeier  &  Klein  (1978) 

-  Cyrus  (1978) 

-  Demaree  et  al.  (1965) 

-  Eddowes  &  Waag  ( 1 980 ) 

-  Huddleston  &  Rolfe  (1971) 

-  Huff  &  Nagel  (1975) 

-  Iff land  &  Whiteside  (1977) 

-  Isley  et  al.  (1974) 

-  Larson  &  Terry  (1975) 

-  Martin  (1981) 

-  Prophet,  Caro  &  Hall  (1972) 

-  Rolfe  &  Waag  (1980) 

-  Roscoe  (1976) 

-  Semple  (1981b) 

-  Valverde  (1973) 

-  Waag  (1981) 

-  Whiteside  (1977) 

-  williges,  Roscoe  &  williges 

(1973) 

-  Woomer  &  Williams  (1978) 


Areas  Other  Than  Flight  Simulation 


-  Adams,  G.  (1977) 

-  Adams,  J.  (1979) 

-  Baer,  Jones  &  Francis  (1980) 

-  Barrett  (1971) 

-  Battelle  (1982) 

-  Beck  &  Monroe  (1969) 

-  Blaiwes  &  Regan  (1970) 

-  Brock  (1978) 

-  Clark  &  Gardner  (1977) 

-  Fink  &  Shriver  (1978a,  1978b) 

-  Freda  (1979) 

-  Gerathewohl  (1969) 

-  Hall,  Lam  &  Bellomy  (1976) 

-  Hall,  Rankin  &  Aagard  (1976) 

-  Hays  (1980,  1981a,  1981b) 

-  Hogan  (1978) 

-  Hopkins  (1975,  1978) 

-  Hritz  et  al.  (1980a,  1980b,  1981, 

1982) 

-  Kinkade  &  Wheaton  (1972) 

-  Knerr,  Simutis  &  Johnson  (In  press) 

-  Malec  (1980) 

-  Micheli  (1972) 

-  Miller,  G.  G.  (1974) 

-  Miller,  K.  E.  (1980) 

-  Montemerlo  (1977) 

-  N.T.I.S.  (1982) 

-  Orlansky  (1981) 

-  Payne  (1982) 

-  Purifoy  &  Benson  (1979) 

-  Semple  (1974b) 

-  Smode  (1971 ,  1972) 

-  Smode  fi  Hall  (1975) 

-  Spangenberg  (1976) 

-  Weitz  &  Adler  (1973) 


Table  3 


Representative  Training  Device  Design  and 
Training  System  Empirical  Studies 


Area  of  Flight  Simulation  Areas  Other  Than  Flight  Simulation 


Motion  Cues 

-  Bray  (1973) 

-  Irish  £  Brown  (1978) 

-  Irish  &  Buckland  (1978) 

-  Jacobs  (1975) 

-  Jacobs  et  al.  (1973) 

-  Koonce  ( 1 979 ) 

-  Martin  &  Waag  (1978a,  1978b) 

-  Nataupsky  et  al.  (1979) 

-  Pohlman  &  Reed  (1978) 

-  Roscoe  &  Williges  (1975) 

-  Ruocco,  Vitale  &  Benfari  (1965) 

-  Ryan,  Scott  £  Browning  (1978) 

-  Woodruff  et  al.  (1976) 

Instrument/Procedures  Training 

-  Bailey,  Hughes  £  Jones  (1980) 

-  Bickley  (1980) 

-  Biersner  (1976) 

-  Brictson  £  Burger  (1976) 

-  Browning  et  al.  (1977) 

-  Burger  &  Brictson  (1976) 

-  Crosby  (1977) 

-  Crosby  et  al.  (1978) 

-  Demaree,  Norman  &  Matheny  (1965) 

-  Ellis  et  al.  (1968) 

-  Goebel,  Baum  &  Hagin  (1971) 

-  Hagin,  Durall  &  Prophet  (1979) 

-  Holman  (1979) 

-  Irish  et  al.  (1977) 

-  Isley  (1968) 

-  Isley  et  al.  (1968) 

-  Krahenbull,  Marett  £  Reid  (1978) 

-  Pitre 11a  (1974) 

-  Prophet  £  Boyd  (1970) 

-  Provenmire  £  Roscoe  (1971,  1973) 

-  Randel  et  al.  (1981) 

-  Reicher  et  al.  (1980) 

-  Reid  £  Cyrus  (1974) 

-  Smith  et  al.  (1974) 

-  Weitzman  et  al.  1979) 


Procedural  Tasks 

-  Bernstein  £  Gonzalez  (1969) 

-  Cox  et  al.  ( 1965) 

-  Crawford  et  al.  (1976) 

-  Grimsley  (1969a,  1969b,  1969c) 

-  Grunwald  (1968) 

-  Johnson  (1978) 

-  McGuirk,  Pieper  £  Miller  (1975) 

-  Mirabella  £  Wheaton  (1973,  1974) 

-  Rigney  et  al.  (1978a,  1978b) 

-  Wheaton  £  Mirabella  (1972a,  1972b) 

-  Wheaton,  Mirabella  £  Farina  ( 1 971 ) 

-  Wright  £  Campbell  (1975) 

Operations  Tasks 

-  Blaauw  (1982) 

-  Finley,  Rheinlander  £  Thompson  (1972) 

-  Goldberg  (1980) 

-  Goldin  £  Thorndyke  (1982) 

-  Gray  (1979) 

-  McKnight  £  Hunter  (1966) 

-  Puig  (1972) 

-  Rose  et  al.  (1976) 

-  Semple  (1974a) 

-  Spangenberg  (1974) 

-  Wagenaar  (1975) 

-  Wheaton  et  al.  (1976c) 

Psychomotor  Tasks 

-  Baum  et  al.  (1982,  In  press) 

-  Reidel,  Abrams  £  Post  (1975) 

Troubleshooting  Tasks 

-  Steinemann  (1966) 

Perceptual  Tasks 

-  Prather  (1971 ) 

-  Prather,  Berry  £  Jones  (1971) 

Avionic  Maintenance 

-  Baum  et  al.  (1979) 

-  Biersner  (1975) 

-  Cicchinelli  (1979) 

-  Cicchinelli  et  al.  (1980,  1982) 

-  Cicchinelli  £  Harmon  (1981) 

-  Goett,  Post  £  Miller  (1980) 


foundation  for  this  systematic  research.  In  this  effort,  physical  and  func¬ 
tional  fidelity  are  separately  manipulated  to  determine  their  relative  con¬ 
tributions  to  training  effectiveness  in  a  psychomotor  task.  This  research 
uses  an  ordinal  measurement  of  physical  and  functional  fidelity  (Hays,  1980). 
Future  research  may  use  these  ordinal  fidelity  metrics  or  develop  other  fidelity 
measures.  Future  efforts  will  also  investigate  other  task  types  in  addition 
to  the  interaction  of  fidelity  with  other  training  system  variables. 

Areas  other  than  military  training  may  provide  insights  into  fidelity 
decisions.  For  example,  the  area  of  sports  training  has  many  useful  procedures 
for  training  tasks  that  may  be  similar  to  tasks  in  the  military  (Rodionov, 

1971).  More  traditional  academic  areas,  such  as  tracking  research  (Poulton, 
1974)  may  also  provide  insights  into  fidelity  decisions.  Experiments  which 
focus  on  tracking  but  which  also  use  devices  of  various  configurations  for 
training  may  be  reanalyzed  in  order  to  gain  new  information  for  making  deci¬ 
sions  about  training  device  fidelity.  Research  which  investigates  these  other 
areas  would  be  a  useful  complement  to  traditional  fidelity  research. 

Instructional  Features  Issues.  During  a  recent  workshop  on  simulator 
fidelity  (Hays,  1981b),  two  divergent  viewpoints  emerged.  One  view  was  that 
fidelity  was  a  very  important  issue  while  the  other  was  that  the  i'.^ortant  issue 
was  what  instructional  features  should  be  incorporated  into  a  training  device. 
Such  instructional  features  might  consist  of  capabilities  such  as  those  listed 
in  Table  4. 

One  instructional  feature  that  deserves  considerable  attention  is  the 
capability  of  providing  augmented  feedback  or  knowledge  of  results.  Feedback 
refers  to  information  presented  to  the  trainee  about  the  adequacy  of  his  or 
her  performance.  Feedback  may  be  used  by  the  trainee  to  guide  ongoing  be¬ 
havior  or  to  improve  the  next  response  (Holding,  1965).  Miller  (1953)  dis¬ 
tinguishes  between  these  two  uses  of  feedback  by  referring  to  them  as  action 
feedback  and  learning  feedback  respectively. 

Holding  (1965)  makes  further  distinctions  between  types  of  feedback. 

The  following  is  based  on  his  discussion.  Feedback  may  be  intrinsic,  present 
in  the  usual  form  of  the  task,  or  artificial  (also  called  augmented),  in  the 
form  of  extra  information  added  in  for  training  purposes.  Furthermore,  aug¬ 
mented  feedback  may  be  concurrent  or  terminal.  Concurrent  feedback  is  infor¬ 
mation  which  is  present  all  the  time  a  person  is  responding,  as  in  watching  a 
pointer  while  adjusting  a  control  knob.  Terminal  feedback  is  information 
which  arises  as  a  result  of  a  completed  response  like  the  score  of  a  dart 
throw.  Likewise,  intrinsic  feedback  may  also  be  either  concurrent  or  terminal. 
Most  of  the  distinctions  shown  in  Figure  3  may  also  be  subdivided.  Only  one 
set  of  branches  is  shown  for  the  sake  of  clarity.  Both  terminal  and  concur¬ 
rent  feedback  may  be  immediate  or  delayed.  These  types  in  their  turn  may  be 
verbal  or  nonverbal.  Either  of  these  types  may  be  given  after  each  response 
as  separate  knowledge,  or  else  accumulated  over  several  attempts  and  presented 
at  the  end  of  the  series.  There  is  a  substantial  body  of  literature  on  the 
effects  of  these  various  types  of  feedback  (see  Holding,  1965  for  a  review). 
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Table  4 


Possible  Instructional  Capabilities  of  Training  Devices 
(Sources:  Miller  et  al.,  1977,  and  Hritz  and  Purifoy,  1982) 


1.  Freeze  Capability,  Under  certain  conditions  such  as  trainee  error,  the 
device  can  freeze  an  instructional  sequence. 

2.  Restart/Resequence  Capability.  The  trainer  has  the  capability  to  restart 
an  instructional  sequence  at  any  point. 

3.  Malfunction  Selection.  The  trainer  can  provide  simulated  malfunctions 
chosen  by  the  instructor. 

4.  Sign-in  Capability.  Trainee  can  sign  in  on  the  device  at  any  authorized 
time  after  providing  specified  information  (passwords,  etc.)  to  the  device. 

5.  Number /Quality  of  Responses.  The  device  can  record,  save,  and  display  both 
the  quality  and  quantity  of  trainee  responses. 

6.  Internal  Monitoring  of  Instructional  Features.  The  device  can  monitor 
specified  variables  and/or  responses  for  specific  actions  (e.g.,  device 
freezes  if  designated  monitor  reads  in  upper  1/2  of  scale  or  device  begins 
providing  altered  feedback  if  a  designated  control  is  activated). 

7.  Augmented  Feedback.  Under  specified  conditions  or  schedules  the  device 
can  enhance  the  feedback  received  by  the  trainee. 

8.  Next  Activity  Features.  Introduction  of  the  next  activity  can  be  linked 
to  specific  trainee  actions  by  the  instructor. 

9.  Stimulus  Instructional  Features.  The  instructor/course  developer  can 
specify  the  rates  and  characteristics  of  stimuli  presented  to  the  trainee. 

10.  Cue  Enhancement  Features.  Device  can  enhance  specified  cues  during 
training  exercises. 

11.  Automated  Demonstration.  Preprogrammed  scenarios  presented  for  trainee 
observation  provide  the  trainee  with  a  model  of  expected  performance  or  the 
consequences  of  some  critical  action. 

12.  Record/Playback .  A  demonstration  technique  that  may  be  used  to  replay 
portions  of  trainee's  behavior.  Often  used  to  review  critical  errors. 


I 
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13.  Adaptive  Syllabus.  Techniques  for  computer  control  of  trainee  pro¬ 
gression  based  upon  performance.  The  training  scenario  is  varied  as  per¬ 
formance  improves. 


FEEDBACK 


INTRINSIC 


ARTIFICIAL 

_ I _ 


LEARNING 


CONCURRENT  TERMINAL 


I - 1 

immediate  delayed 


ACTION 


I  I 

NONVERBAL  VERBAL 


SEPARATE  ACCUMULATED 


Figure  3.  Different  kinds  of  feedback  (knowledge  of  results). 
(Source:  Holding,  1965) 


Boldovici  (1983)  discusses  the  dimensions  of  feedback  in  terms  of  changes 
in  the  salience  of  stimuli.  He  characterizes  feedback  as  a  means  of  altering 
the  salience  of  the  stimuli  that  initiate  performance  or  increase  the  potential 
reinforcing  value  of  stimuli  that  maintain  performance.  Boldovici  first  dis¬ 
tinguishes  between  augmenting  and  supplementing  feedback. 

Augmenting  means  increasing  the  value  of  a  dimension  of  the  signal-- 
its  size,  brightness,  or  frequency  of  appearance.  Supplementing 
means  adding  a  new  dimension  to  the  signal;  a  flashing  light  in¬ 
dicating  that  the  main  gun  safety  switch  is  in  the  Fire  position 
is  an  example.  (Boldovici,  1983) 

Boldovici  then  discusses  how  feedback  may  be  altered  either  by  attenuating  or 
masking  the  noise  in  which  it  is  embedded. 

Attenuating  means  decreasing  the  value  of  a  dimension  of  the  noise — 
its  size,  brightness,  or  frequency  of  appearance  for  example.  At¬ 
tenuating  is  essentially  the  opposite  of  augmenting,  applied  to 
noise.  Masking  means  adding  a  new  dimension  to  the  noise.  Tones 
introduced  into  the  ears  of  people  with  tinnitus,  for  the  purpose 
of  reducing  or  cancelling  "ringing"  exemplify  masking  (Boldovici, 

1983). 


The  main  problem  with  using  any  of  these  techniques  in  training  is  that 
they  may  produce  a  "crutch  effect"  when  transferring  from  the  practice  to  the 
criterion  environment.  In  other  words,  the  trainee  may  come  to  depend  on  the 
altered  stimuli  in  the  training  environment  and  then  when  the  criterion  en¬ 
vironment  does  not  provide  such  altered  stimuli,  performance  may  suffer.  One 
means  of  overcoming  the  problems  associated  with  the  crutch  effect  is  to 
gradually  fade  the  altered  stimuli  during  the  course  of  training  until  cri¬ 
terion  stimuli  levels  are  reached. 

Research  is  needed  to  determine  the  appropriate  times  to  use  altered 
stimulus  salience  in  training  and  also  when  and  how  to  minimize  crutch  ef¬ 
fects.  Boldovici  (1983)  reviews  some  of  the  research  bearing  on  these  ques¬ 
tions.  Research  is  also  needed  to  investigate  the  training  effects  of  other 
types  of  instructional  features.  Such  data  will  enable  training  equipment 
designers  to  include  only  those  instructional  features  that  facilitate  transfer 
of  training  and  exclude  features  that  have  only  cosmetic  appeal  but  do  little 
to  enhance  training  effectiveness. 

Training  Device  Architectural  Issues.  Data  are  needed  on  the  effects  of 
specific  architectural  features  of  training  devices.  For  example,  research 
could  be  conducted  contrasting  the  training  effectiveness  of  hands-on  practice 
using  a  3D  versus  a  2D  medium.  A  3D  practice  medium  would  allow  the  trainee 
to  exercise  the  motor  skills  involved  in  the  tasks  while  a  2D  practice  medium 
(such  as  a  CRT  touch  panel)  would  only  afford  practice  on  the  functional  as¬ 
pects  of  the  task.  Cicchinelli  and  Harmon  (1981)  made  performance  and  cost 
comparisons  among  a  3D  simulator,  a  2D  simulator,  and  the  operational  6883 
F"—  111  Converter/Flight  Control  System  maintenance  test  station.  No  signifi¬ 
cant  differences  in  student  performance,  as  a  function  of  training  device 
employed,  were  found.  However,  actual  equipment  costs  were  found  to  be  twice 
as  high  as  for  the  3D  simulator.  The  relative  cost  effectiveness  of  the  2D 
and  3D  simulators  were  not  known  at  the  time  the  article  was  written.  Decisions 
on  which  type  of  "hands-on"  practice  medium  to  incorporate  into  a  training 
device  should  be  based  on  this  kind  of  empirical  data  on  the  effectiveness  of 
different  simulator  architectures. 

Another  architectural  feature  which  requires  an  experimental  assessment 
to  determine  its  effectiveness  is  the  mix  of  student  and  instructor  stations. 

Is  it  more  training-effective  to  have  one  student  station  per  instructor  sta¬ 
tion  or  can  one  instructor  station  monitor  several  student  stations?  Related 
to  this  question  is  whether  more  than  one  student  should  interact  at  a  student 
station  at  one  time.  Answers  to  these  questions  will  provide  not  only  training 
device  design  guidance  but  also  guidance  in  the  development  of  programs  of 
instruction  (POIs). 

Still  another  architectural  feature  issue  is  the  method  of  trainee-device 
interaction.  Seibel  (1972)  discusses  a  wide  variety  of  alternative  means  for 
entering  data,  These  range  from  keyboards,  levers,  switches  and  dials,  to 
light  pencils,  and  handwritten  or  voice  inputs.  The  choice  of  when  to  use  a 
specific  type  of  input  device  depends  on  (a)  the  characteristics  of  the  data 
to  be  entered,  (b)  the  design  of  the  data  entry  device  and  (c)  the  character¬ 
istics  of  the  operator.  Although  Seibel' s  discussion  is  not  about  training 
devices,  the  same  considerations  apply  in  the  training  environment.  Data  on 
the  training  effectiveness  of  each  possible  input  medium  are  required  before 
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informed  decisions  about  the  incorporation  of  a  specific  trainee  data  input 
feature  into  a  training  device  can  be  made. 


Training  Device  Implementation  Issues 

Any  training  device,  no  matter  what  its  design,  must  be  used  if  it  is  to 
provide  training.  An  important  aspect  of  the  ISD  process  is  therefore  the 
implementation  (installation  and  use)  of  the  training  device.  The  first  set 
of  issues  in  this  context  focuses  on  the  general  acceptance  of  a  training 
device. 

User  Acceptance  Issues.  No  training  device  can  provide  effective  train¬ 
ing  if  it  is  not  accepted  and  used  by  an  instructor.  There  are  cases  in  which 
training  devices  are  not  used  or  are  underutilized  (Caro,  Shelnutt  &  Spears, 

1981;  Mackie,  et  al.  1972).  There  are  also  cases  in  which  the  instructional 
features  of  a  device  are  not  used  (Semple,  1981a).  Often  this  lack  of  instructor 

acceptance  or  use  can  be  attributed  to  the  lack  of  fidelity  in  the  device. 

Instructors  know  the  real  equipment  better  than  they  know  instructional  tech¬ 
niques  and  therefore  may  misjudge  a  device  if  it  does  not  have  "enough"  fidelity. 

In  other  cases,  the  instructor  is  not  fully  trained  on  the  instructional  features 
of  the  device.  This  point  is  made  by  Caro,  Shelnutt  and  Spears  (1981)  when  they 
draw  the  distinction  between  acceptance  of  a  device  as  it  is  used  and  acceptance 
of  a  device  as  it  was  designed  to  be  used. 

Only  a  few  studies  have  addressed  the  acceptance  issue  (Biersner,  1975; 
Stoffer  et  al.,  1980).  These  studies,  although  acknowledging  the  importance  of 
user  acceptance,  do  not  systematically  study  the  training  effects  of  nonaccept¬ 
ance.  It  is  therefore  essential  that  additional  research  be  conducted  to  de¬ 
termine  all  the  relevant  dimensions  of  instructor  acceptance  of  a  training 
device.  Once  these  dimensions  are  determined,  specific  instructor  training 
procedures  can  be  developed  to  aid  in  the  operational  installation  of  a  new 
training  device.  An  instructor  training  procedure  to  counteract  specific 
misperceptions  about  training  devices  (e.g.,  that  high  fidelity  is  essential) 
would  do  much  to  improve  the  ease  and  efficiency  with  which  a  training  device 
is  implemented  and  used. 

Analogous  to  instructor  acceptance  is  the  issue  of  training  device  accept¬ 
ance  by  trainees.  Though  trainees  do  not  have  the  experience  with  actual 
equipment  that  instructors  do,  they  still  need  a  certain  degree  of  realism  to 
motivate  their  training.  Bleda  (1979)  reports  that  trainees  overall  job 
satisfaction  and  job-related  motivation  may  be  improved  by  a  well  designed 
training  program.  Research  is  necessary  to  determine  the  dimensions  of  trainee 
acceptance  of  training  devices  just  as  it  is  necessary  in  the  area  of  instructor 
acceptance.  Before  device  design  specifications  are  finalized,  input  concerning 
device  acceptance  should  be  incorporated  into  the  process  of  determining  those 
specifications . 

t 

Training  Device  Integration  Issues.  In  addition  to  instructors  and 
trainees  accepting  a  training  device,  it  is  necessary  for  the  device  to  be 
integrated  into  an  ongoing  POI.  The  process  of  integrating  the  training 
device  may  not  always  be  an  easy  one.  Often  courses  are  designed  to  train 
groups  of  trainees  while  the  training  device  is  designed  to  train  one  trainee 
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at  a  time.  It  may  be  necessary  to  design  the  training  device  to  accommodate 
this  team  or  group  training  approach.  On  the  other  hand,  if  a  device  proves 
to  be  more  effective  with  one  trainee  at  a  time,  it  may  be  more  beneficial  to 
modify  the  existing  POI  to  take  advantage  of  the  training  device.  Systematic 
research  and  coordination  is  necessary  to  determine  the  most  efficient  process 
for  facilitating  device  integration. 

As  with  other  issues  already  discussed,  the  issue  of  device  integration 
is  an  interactive  one.  Device  design  characteristics,  instructor  and  trainee 
acceptance  of  the  device,  analysis  of  what  must  be  trained  as  well  as  analysis 
of  current  POIs  all  contribute  to  the  ultimate  decisions  of  how  to  integrate 
the  training  device  into  an  instructional  context. 

Noninstitutional  Use  of  Training  Devices.  One  of  the  goals  of  the  Army 
Maintenance  Training  and  Evaluation  Simulation  System  (AMTESS)  is  to  provide 
both  institutional  (school)  training  and  also  refresher  training  and  skill 
evaluation  in  the  field  (Dybas,  1981).  This  is  a  new  approach  to  training 
device  utilization  and  basic  data  must  be  generated  to  determine  how  such  de¬ 
vice  use  is  to  proceed.  Data  would  be  useful  on  at  least  the  following  issues: 

o  differences  between  tasks  in  the  institution  and  in  the  field 

o  types  of  refresher  training  necessary 

o  device  acceptance 

o  integration  of  device  into  a  field  context 

o  evaluation  of  device  effectiveness  as  field  tool. 

Once  we  have  these  necessary  data  on  noninstitutional  training  device  utiliza¬ 
tion,  the  Army  will  be  in  a  better  position  to  implement  field  operations  both 
efficiently  and  effectively. 


Issues  in  Training  Effectiveness  Assessment 

The  only  remaining  major  section  of  the  ISD  model  to  be  discussed  is  the 
area  of  training  effectiveness  assessment.  Although  we  discuss  this  area 
last,  in  terms  of  an  already  existing  POI,  the  assessment  of  training  ef¬ 
fectiveness  must  be  the  first  step.  The  relative  training  effectiveness  of  any 
POI  must  be  assessed  before  it  may  be  either  modified,  if  it  is  not  training 
effective,  or  copied,  if  the  program  is  proven  to  be  training  effective.  The 
need  for  ongoing,  adequate  assessment  cannot  be  overemphasized.  When  every 
training  course  in  the  military  is  adequately  monitored,  then  optimal  adjust¬ 
ments  may  be  made  to  deal  with  any  changes  due  to  trainee  characteristics, 
media  and  device  utilization,  training  strategies,  or  task  criteria.  These 
adjustments  may  then  afford  the  military  the  opportunity  to  maintain  the 
training  effectiveness,  efficiency,  and  quality  of  its  POIs  at  the  highest 
possible  level. 

There  are  several  possible  approaches  to  the  assessment  of  training  ef¬ 
fectiveness  and  several  major  issues  to  be  considered  during  the  selection  of 
the  appropriate  assessment  method(s)  and  types  of  measures.  Table  5  lists 
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Table  5 


Representative  Publications  which  Discuss  Training  Effectiveness 
Analysis  Methodology  Issues 


-  Anderson,  Ball,  Murphy  &  Rosenthal  (1973) 

-  ATSC  (1980) 

-  Blaiwes,  Puig  &  Regan  (1973) 

-  Boldovici  (1981) 

-  Caro  (1970,  1973a,  1977) 

-  Evaluation  of  Training  (1981) 

-  Hill  &  Kress  (1979) 

-  Jeantheau  (1971) 

-  Klein,  Kane,  Chinn  &  Jukes  (1978) 

-  Leonard,  Wheaton  &  Cohen  (1975) 

-  Naval  Training  Device  Center  (1966) 

-  Payne  (1982) 


several  representative  publications  which  discuss  methodological  issues  in 
training  effectiveness  assessment.  No  single  approach  or  measure  is  empha¬ 
sized  in  this  report  because  each  approach  has  its  strengths  and  its  weak¬ 
nesses.  These  positive  attributes  as  well  as  the  problems  associated  with 
each  approach  must  be  weighed  before  making  the  decision  to  use  any  one 
assessment  technique  or  mixture  of  techniques.  The  first  part  of  this  sec¬ 
tion  is  a  review  of  general  issues  in  evaluation  methods  and  measures.  In 
the  second  part  of  this  section,  different  types  of  training  effectiveness 
measures  are  reviewed,  their  strengths  and  weaknesses  compared,  and  a  few 
examples  of  each  type  of  measure  are  presented.  The  third  part  of  this 
section  is  a  discussion  of  the  establishment  of  appropriate  performance 
standards  and  the  measures  and  methods  used  to  determine  if  training  ef¬ 
fectively  meets  these  standards.  The  final  portion  of  this  section  is  a 
discussion  of  the  resource  requirements  necessary  to  support  the  evaluation 
of  training  effectiveness. 

General  Training  Evaluation  Measurement  Issues.  Before  an  instructor, 
program  director,  or  researcher  can  evaluate  a  training  program,  much  less 
compare  that  program  to  a  set  of  standards  or  to  another  program,  the  training 
effectiveness  of  the  program  must  first  be  measured.  That  measurement  must  be 
relevant,  accurate,  and  valid,  or  the  entire  procedure  is  a  waste  of  time  and 
money.  There  are  three  general  issues  in  the  basic  methods  of  evaluation. 

The  first  issue  concerns  when  the  training  effectiveness  is  measured.  The 
second  concerns  how  the  training  effectiveness  is  measured.  The  third  issue 
concerns  the  validity  of  the  measures  that  are  used. 

Training  effectiveness  measures  can  be  divided  into  two  general  classes, 
process  and  outcome  measures  (Goldstein,  1974).  The  continual  measuring  of 
progress  during  training,  for  example,  after  the  completion  of  instruction  on 
a  single  task  in  a  multitask  training  program,  is  called  process  measurement. 
The  general  purpose  of  process  measures  is  to  ensure  that  program  goals  are 
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being  met  in  a  timely  fashion  and  that  the  knowledges  and  skills  needed  for 
later  training  are  being  adequately  learned.  These  measures  are  usually  single 
task  or  single  skill  measures  and  are  usually  criterion- referenced,  objective 
type  measures  (these  terms  will  be  explained  below).  Outcome  measures  are  ap¬ 
plied  after  the  completion  of  the  training  program,  whether  the  program  is 
simple  and  brief  or  complicated  and  time  consuming.  These  measures  are  attempts 
to  evaluate  the  training  effectiveness  of  the  entire  POI.  They  can  occur  im¬ 
mediately  after  training,  when  the  trainee  is  first  applying  the  new  knowledge 
and  skills  on  the  actual  job.  In  this  case  they  have  been  called  proximal 
measures  (Goldstein,  1974).  If  measures  are  taken  after  some  long  period  of 
time  has  elapsed  on  the  job,  they  are  called  distal  measures  (Goldstein,  1974). 
The  proximal  measures  are  typically  used  in  transfer  of  training  (TOT)  studies. 
Distal  measures  are  used  to  evaluate  long-term  retention,  typically  with  skills 
or  knowledges  that  do  not  get  used  very  often.  Obviously,  the  decision  to  use 
one  type  of  measure  must  be  based  on  what  kind  of  information  is  needed  in  the 
evaluation. 

The  choice  of  when  to  conduct  the  training  effectiveness  evaluation  can 
introduce  biases,  which  can  influence  the  results  of  the  evaluation.  One  ex¬ 
ample  of  these  biases  comes  from  research  in  the  area  of  learning  and  reten¬ 
tion.  Atkinson  (1972)  used  several  different  instructional  strategies  in 
teaching  German  words  while  investigating  learning  and  retention.  One  strategy 
presented  the  paired  English  and  equivalent  German  words  randomly  on  a  computer 
screen.  Another  strategy  allowed  the  learner  to  select  the  pairs  to  be  prac¬ 
ticed.  The  third  strategy  was  under  the  control  of  computer  algorithms  that 
were  designed  to  present  items  until  a  criterion  was  reached  and  to  optimize 
the  number  of  word  pairs  learned  to  that  criterion.  A  process  measure  of 
retention,  based  on  the  number  of  errors  made,  found  that,  in  terms  of  reten¬ 
tion,  the  random  method  was  best  and  the  computer  algorithms  method  was 
worst.  However,  shortly  after  training,  a  proximal,  outcome  measure  found 
the  situation  reversed.  The  implication  is  that  both  methods  should  be  used 
whenever  practical  and  that  the  results  generated  by  the  methods  should  be 
compared  and  evaluated  before  any  conclusions  about  the  effectiveness  of 
the  POI  are  drawn. 

When  determining  how  to  evaluate  training  effectiveness,  several  issues 
must  be  considered.  The  first  issue  concerns  whether  the  measure  is  based  on 
objective  criteria  or  whether  it  is  based  on  subjective  criteria.  Objective 
criteria  are  quantifiable  things  such  as  rate  of  production,  number  of  correct 
or  incorrect  answers,  etc.  A  problem  with  objective  criteria  lies  in  the  in¬ 
dividual  differences  of  the  trainees.  The  previous  experiences  or  higher  ap¬ 
titudes  of  certain  trainees  may  mask  the  training  effect  of  the  POI  and  thus 
invalidate  the  measure.  At  the  other  end  of  this  dimension  are  measures 
based  on  subjective  criteria.  Subjective  based  measures  commonly  use  peer 
ratings  or  instructor  evaluations  to  judge  the  effectiveness  of  the  training 
course.  The  greatest  problem  with  subjective  criteria  is  that  human  estima¬ 
tion  is  very  easily  biased  and  may  invalidate  the  measure.  Care  should  be 
taken  to  insure  that  a  proper  mix  of  objective  and  subjective  criteria  are 
used  in  any  program  evaluation.  Research  on  the  appropriate  mix  of  these 
two  approaches  in  various  evaluation  applications  would  provide  a  basis  for 
training  evaluators  to  make  informed  decisions  about  which  type(s)  of  criteria 
to  use  in  future  evaluations. 
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Another  issue  in  the  measurement  of  training  effectiveness  concerns  how 
the  effectiveness  measures  are  referenced.  Performance  measures  can  be  either 
criterion- referenced  or  norm-referenced  (Goldstein,  1974).  Criterion- referenced 
measures  are  tied  to  absolute  standards,  generally  using  specific  objective 
behavioral  items,  although  subjective  items  could  be  used.  Norm- referenced 
measures  simply  evaluate  the  trainees'  standing  vis  a  vis  their  peers,  a 
naive  group,  or  possibly  a  group  of  experts.  The  best  information  for  evalua¬ 
tion  of  training  can  be  acquired  by  using  objective,  criterion- referenced 
measures.  The  least  valuable  information  for  evaluation  of  training  would 
come  from  subjective  ratings  that  are  only  referenced  to  the  trainee  group. 

All  that  would  be  known  in  this  latter  case  is  how  the  instructor  thinks  the 
trainees  perform  with  reference  to  their  classmates,  not  how  they  rate  with 
reference  to  actually  learning  the  required  skills  in  the  POI. 

The  last  general  issue  in  methods  of  measurement  concerns  validity.  En¬ 
suring  that  the  measures  used  in  training  effectiveness  evaluation  are  valid 
is  probably  the  most  difficult  part  of  any  evaluation.  There  are  two  general 
classes  of  validity,  internal  validity  and  external  validity. 

Internal  validity  refers  to  the  approximate  validity  with  which 
we  infer  that  a  relationship  between  two  variables  is  causal  or 
that  the  absence  of  a  relationship  implies  the  absence  of  cause. 

External  validity  refers  to  the  approximate  validity  with  which 
we  can  infer  that  the  presumed  causal  relationship  can  be  gen¬ 
eralized  to  and  across  alternate  measures  of  the  cause  and  effect 
and  across  different  types  of  persons,  settings,  and  times 
(Cook  &  Campbell,  1979,  p.  37). 

Internal  validity  is  concerned  with  the  structure  of  the  experiment.  If  proper 
controls  are  not  used  and  if  proper  statistical  interpretations  are  not  made, 
then  there  can  be  no  faith  that  a  true  causal  relationship  exists  between  the 
variables  of  interest.  External  validity,  on  the  other  hand,  is  concerned 
with  whether  the  experimentally  demonstrated  relationship  can  be  regarded  as 
an  example  of  a  relationship  that  could  also  be  found  outside  of  the  laboratory 
and  whether  the  experiment  actually  measured  that  "real  world"  relationship. 
External  validity  is  a  matter  of  inference.  In  the  case  of  a  training  ex¬ 
periment  the  major  focus  of  external  validity  is  the  generalizability  of  the 
information  about  the  particular  training  situation  to  some  other  situation 
or  to  training  in  general.  Questions  about  external  validity  center  on  how 
representative  the  data  are  of  the  general  class  of  training  procedures.  The 
evaluation  of  internal  and  external  validity  will  be  one  of  the  primary  methods 
used  to  sort  out  and  evaluate  previous  research  on  training  programs  for  inclu¬ 
sion  in  the  training  research  issues  data  base. 

Types  of  Training  Effectiveness  Assessment.  There  are  many  possible  ap¬ 
proaches  to  the  assessment  of  training  effectiveness.  All  of  these  approaches 
require  some  use  of  proficiency  measures,  with  all  of  the  problems  discussed 
above.  Once  the  type  of  proficiency  measure  is  selected,  the  next  question 
is  how  to  apply  the  proficiency  information  to  assess  the  effectiveness  of 
training.  One  highly  acclaimed  and  strongly  advocated  approach  is  transfer 
of  training  (Orlansky, . 1981 ;  Orlansky  &  String,  1981a).  In  general,  the 
transfer  of  training  (TOT)  paradigm  is  based  on  the  effect  a  previously 
learned  skill  has  on  the  acquisition  or  relearning  of  some  second  skill. 


27 


There  are  several  different  models  of  TOT  and  various  derived  designs,  loosely 
based  on  TOT  assumptions,  that  can  be  used  in  training  effectiveness  assess¬ 
ment  (Caro,  1977).  A  different  approach  to  evaluating  training  effectiveness 
is  to  base  the  overall  conclusions  about  effectiveness  on  the  comparative 
costs  of  different  training  devices  (Puig,  1972;  King,  1978).  Ttie  common 
ground  for  both  of  these  general  approaches  is  that  adequate  measures  of 
proficiency  are  required  first. 

As  illustrated  earlier  in  Figure  2,  the  assessment  of  training  effective¬ 
ness  must  weigh  the  various  external  threats  against  the  quality  of  instruc¬ 
tion  in  an  existing  POI.  This  is  usually  accomplished  by  using  proficiency 
tests  already  developed  in  the  school.  The  adequacy  of  these  proficiency 
tests  is  crucial  because,  short  of  war,  there  are  few  other  methods  for 
determining  the  quality  of  Army  training  and  thus  overall  force  readiness. 

As  discussed  above,  there  are  many  problems  associated  with  the  process  of 
proficiency  evaluation  test  development,  especially  in  the  case  of  training 
device  evaluation.  It  may  be  necessary  to  modify  existing  proficiency  tests 
in  order  to  highlight  specific  questions  about  the  design  of  a  training  de¬ 
vice  or  its  use  in  a  POI.  Also,  based  on  the  issues  discussed  above,  it  may 
be  possible  to  modify  proficiency  tests  in  order  to  provide  more  reliable 
and  valid  data  on  training  effectiveness.  In  any  case,  the  processes  in¬ 
volved  in  test  development  and  implementation  should  be  identified  and 
cataloged  for  more  systematic  use  and  in  general  to  maximize  the  efficiency 
of  proficiency  testing  across  the  Army's  MOSs. 

There  are  three  major  issues  which  must  be  resolved  during  the  selection 
of  a  method  for  training  effectiveness  assessment.  The  first  issue  to  be 
addressed  is  whether  to  assess  training  effectiveness  by  measuring  trainee 
proficiency  or  to  evaluate  the  training  device  and/or  POI  on  some  other  basis. 
For  example,  there  are  several  general  research  designs  (to  be  discussed  be¬ 
low)  which  assess  training  effectiveness  by  using  subject  matter  expert  opinions 
rather  than  using  some  measure  of  trainee's  actual  performance.  The  second 
issue  is  whether  to  choose  time  in  training  or  performance  quality  as  the  basic 
effectiveness  measure.  For  example,  a  specific  training  device  or  POI  may  not 
improve  the  final  level  of  performance,  but  may  provide  an  adequate  level  of 
proficiency  in  less  time.  On  the  other  hand,  a  training  regimen  or  training 
device  may  require  slightly  more  training  time,  but  may  greatly  improve  per¬ 
formance  over  the  long  run.  The  third  issue  concerns  when  to  measure  train¬ 
ing.  Is  training  effectiveness  measured  over  the  short  term  or  are  the 
measures  based  on  long-term  retention?  Most  training  effectiveness  studies  are 
conducted  in  the  school  during  or  shortly  after  training  and  can  provide  little 
information  about  long-term  trainee  performance  in  the  field.  Empirical  data 
about  these  three  issues  will  facilitate  management  decisions  about  the 
implementation  of  training  effectiveness  assessment  since  decisions  concerning 
these  issues  must  be  made  before  specific  research  designs  may  be  implemented. 

The  best  recent  review  of  research  designs  for  determining  training  device 
and  training  program  effectiveness  is  by  Caro  (1977).  A  detailed  breakdown 
of  Caro's  ten  distinct  designs  is  presented  in  Table  6.  The  ten  approaches 
can  be  grouped  into  four  categories:  the  classic  TOT  design  and  three  groups 
of  designs  which  differ  on  various  dimensions  from  the  classic  TOT  design. 

The  classic  TOT  design  will  be  described  first  and  then  the  other  three 
groups  of  designs  will  be  discussed. 
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The  classic  transfer  of  training  design  is  based  on  two  assumptions. 

The  first  assumption  is  that  the  mastery  of  one  skill  will  affect  the  acqui¬ 
sition  of  another  similar  skill.  The  second  assumption  that  the  best  way  to 
evaluate  the  effect  of  a  training  device  is  to  compare  the  training  outcome 
to  a  matching  training  program  that  doesn't  use  that  training  device  (i.e., 
uses  some  other  training  device).  The  object  of  this  relative  evaluation  is 
to  develop  the  training  device,  or  more  generally  the  overall  training  pro¬ 
gram,  that  leads  to  the  best  performance  on  the  actual  equipment  both  im¬ 
mediately  and  over  the  long  term. 

As  can  be  seen  from  the  top  line  in  Table  6,  and  as  implied  above,  a 
classic  TOT  study  involves  two  groups  which  are  treated  in  almost  exactly  the 
same  way.  The  control  or  baseline  group  goes  through  the  training  program 
without  the  training  device  of  interest,  using  either  another  training  device 
or  the  actual  equipment  for  training.  The  experimental  or  comparison  group 
goes  through  the  same  training  program  using  the  training  device  of  interest. 
(Note  that  the  same  pattern  would  hold  for  evaluating  any  change  made  in  the 
overall  program  with  the  intent  of  improving  the  program.)  The  two  groups 
are  evaluated  in  the  same  way  at  the  end  of  training  and  their  proficiency  is 
compared  in  order  to  determine  the  relative  training  effectiveness  of  the  new 
training  device  or  training  program. 

The  best  comparative  measures  are  to  some  extent  based  on  the  general 
TOT  model.  Two  of  the  best  known  measures  can  be  based  on  either  performance 
levels  or  time  in  training.  A  percent  transfer  measure  can  be  calculated  to 
determine  what  increase  in  efficiency,  expressed  in  percent  relative  to  the 
control  or  baseline  group,  is  found  in  the  experimental  or  new  training  group 
(see  formula  1).  In  this  formula,  Tc  refers  to  either  the  time  in  training 
or  to  the  proficiency  level  of  the  control  group  and  Te  is  the  time  in  train¬ 
ing  or  the  proficiency  level  of  the  experimental  group  (Micheli,  1972). 


%  transfer  =  _  x  100  (1) 


Another  measure  that  may  be  used  with  the  TOT  model  is  the  transfer  effective¬ 
ness  ratio  (TER).  This  ratio  is  recommended  because  it  centers  on  the  time 
or  trials  to  some  performance  criterion  after  the  shift  to  actual  equipment 
conditions  and  therefore  is  more  directly  relevant  to  long  term,  on  the  job 
performance  rather  than  just  training  school  proficiency.  In  the  TER  formula 
(see  formula  2),  Tc  refers  to  time  or  trials  spent  on  the  actual  equipment 
by  the  control  group,  Te  is  the  same  measure  for  the  experimental  group  on 
the  actual  equipment  and  Xe  refers  to  the  same  or  a  similar  measure  for  the 
experimental  group  on  the  training  device  (Provenmire  &  Roscoe,  1973). 


TER  = 


(2) 


30 


The  level  of  performance  is  the  same  for  both  groups  and  therefore  the  ratio 
can  be  directly  related  to  underlying  cost  variables  allowing  the  most  effec¬ 
tive  level  of  effort  for  reaching  an  established  criterion.  Use  of  the  TER 
encourages  delayed  measures  rather  than  immediate  ones  and  may  therefore  be 
more  indicative  of  long-term  behavior  (Atkinson,  1972).  These  measures  are 
based  on  at  least  close  approximations  of  the  TOT  model  and  are  difficult  to 
use  in  other  types  of  evaluation  designs. 

The  first  group  of  models  derived  from  the  classic  TOT  design  differ  in 
the  way  in  which  they  deal  with  the  control  or  baseline  groups.  This  group 
of  models  consists  of:  the  self-control  model,  the  pre-existing  control 
model,  and  the  uncontrolled  model  (see  Table  6).  The  self-control  model  uses 
a  pretest  (a  before  training  test  on  the  proficiency  measure)  as  the  basis  of 
comparison  for  the  training  program.  The  lack  of  a  control  group  presents  a 
problem  in  evaluation,  since  it  would  be  a  strange  phenomenon  indeed  if  no 
improvement  was  found  between  the  first  test  and  the  second.  Whether  to 
attribute  the  increase  in  proficiency  to  the  training  device  used  in  the 
training  program  or  to  some  other  change  in  the  subjects  becomes  an  insurmount¬ 
able  problem.  The  pre-existing  control  model,  which  uses  pre-existing 
information,  such  as  trainee  scores  from  previous  groups,  may  seem  to  alleviate 
this  problem.  However,  comparisons  to  previous  trainee  groups  presents  other 
equally  serious  problems.  The  essence  of  the  control  group  is  that  they  are 
given  the  same  treatment  as  the  experimental  group,  with  the  exception  of  the 
training  device  experience.  When  comparing  to  previous  groups,  the  assumed 
commonality  may  not  exist.  Any  change  in  the  POI,  the  instructors,  the 
trainees,  or  the  outcome  measures  may  invalidate  the  comparison  because 
performance  differences,  if  any,  may  be  due  to  factors  other  than  the  train¬ 
ing  device.  Both  of  these  types  of  problems,  possible  interfering  factors 
and  the  inability  to  specify  proficiency  increases  due  to  training,  are 
compounded  in  the  uncontrolled  model.  The  major  unanswered  question  after 
an  evaluation  based  on  the  uncontrolled  design  is  whether  or  not  the  level 
of  proficiency  would  have  been  achieved  without  training.  The  assumption 
that  because  certain  criteria  are  met  the  training  device  and  training  pro¬ 
gram  must  be  effective,  is  completely  unwarranted. 

The  next  group  of  evaluation  models  suffers  due  to  multiple  differences 
from  the  TOT  model.  This  group  includes:  the  training  device- to-training 
device  model  and  the  training  device  performance  improvement  model  (see  Table  6). 
In  the  training  device-to-training  device  model,  no  control  group  is  used  and 
the  assumption  is  that  the  final  measure  (on  the  new  training  device)  is 
equivalent  to  the  performance  proficiency  that  would  be  found  on  the  actual 
equipment.  In  the  training  device  performance  improvement  model,  the 
assumptions  are  the  same  as  those  made  in  the  self-control  transfer  model, 

that  if  the  trainees  evidence  any  improvement  then  the  training  has  been 

effective.  The  major  problem,  in  addition  to  the  lack  of  a  control  group,  is 

that  there  is  no  real  reference  to  the  actual  equipment.  Both  of  these  models 
attempt  to  measure  training  rather  than  to  evaluate  the  training  device  or 
training  program  and  generally  use  performance  measures  in  a  noncomparative 
fashion.  Since  the  models  do  not  include  references  to  actual  equipment,  the 
measures  are  all  immediate  and  the  information  level  for  predicting  competence 
in  field  work  is  low. 
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The  last  group  of  models  are  so  different  from  the  classical  TOT  model 
that  there  is  very  little  basis  for  comparison.  This  group  consists  of:  the 
backward  transfer  model,  the  training  device  fidelity  model,  the  training 
device  program  analysis  model,  and  the  opinion  survey  model  (see  Table  6). 

In  the  backward  transfer  model,  an  expert  is  tested  on  the  training  device. 

The  assumption  is  that  if  the  expert  can  adequately  transfer  skills  and 
proficiency  from  actual  equipment  to  the  training  device  then  a  trainee  on 
the  device  should  be  able  to  adequately  transfer  his  or  her  skills  to  the 
actual  equipment.  While  the  assumption  that  transfer  works  the  same  way  in 
both  directions  may  seem  reasonable,  there  has  been  no  empirical  verification 
of  this  assumption. 

The  training  device  fidelity  model  is  based  on  the  assumption  that  it 
the  training  device  closely  approximates  the  actual  equipment  then  its  train¬ 
ing  effectiveness  must  be  high.  The  Army  has  sponsored  the  development  of 
several  predictive  models,  collectively  known  as  TRAINVICE,  which  allow  the 
training  developer  to  analytically  assess  the  effectiveness  of  a  training 
device  and  to  generate  a  figure  of  merit  for  each  device  configuration 
(Wheaton  et  al.  1976a,  1976b,  1976d;  Narva,  1979a,  1979b;  PM  TRADE,  197y; 
Swezey  &  Evans,  1980;  Swezey  &,  Narva,  unpublished).  Tufano  and  Evans  (in 
press)  reviewed,  compared,  and  contrasted  the  four  TRAINVICE  models.  Their 
recommendations  require  research  to  be  done  in  two  areas:  field  validation 
of  the  TRAINVICE  models  in  various  task  domains  and  longer  range  investiga¬ 
tions  of  the  model's  underlying  assumptions. 

The  training  device  program  analysis  model  assumes  that  training  will 
be  effective  if  the  POI  is  well  designed  and  uses  good  instructional  tech¬ 
niques.  Both  the  training  device  program  analysis  model  and  the  training 
device  fidelity  model  are  limited  in  that  they  only  evaluate  the  training 
device  or  program  in  an  analytic  manner.  Neither  approach  makes  use  of  actual 
training  behavior  as  assessed  by  either  time-based  or  performance-based  meas¬ 
ures  and  neither  provides  information  about  the  resultant  transfer  of  skills 
from  the  training  device  to  the  actual  equipment.  This  is  also  true  of  the 
opinion  survey  model.  The  data  for  this  model  consist  of  opinions  about  the 
POI  and  the  training  device  obtained  from  equipment  operators,  course  devel¬ 
opers,  instructors,  and  even  trainees.  To  the  extent  that  these  "experts" 
are  knowledgeable  about  training  objectives  and  methods,  their  opinions  may 
be  of  value.  However,  this  process  is  more  prone  to  error  than  a  formal 
analysis  of  either  the  training  device  fidelity  or  the  content  of  the  POI 
and  does  not  provide  any  substantial  information  about  training  effectiveness. 


Assessing  Cost  Effectiveness 

To  reemphasize  a  point  made  by  many  authors  (e.g.,  Lawrence,  1954),  cost 
is  often  the  ultimate  criterion  against  which  training  is  judged.  As  noted 
in  the  introduction  to  this  section,  an  adequate  cost  effectiveness  appraisal 
must  be  based  on  an  adequate  training  effectiveness  assessment.  The  position 
taken  in  this  paper  is  that  the  efficient  and  correct  use  of  the  ISD  process 
requires  the  use  of  an  adequate  empirical  assessment  of  training  effectiveness 
that  can  lead  directly  to  reasonable  cost  analyses. 


32 


One  popular  method  of  cost  analysis,  that  is  not  always  based  on  the  use 
of  adequate  assessments  of  training  effectiveness,  is  a  simple  one-to-one 
cost  comparison  of  training  devices  and/or  actual  equipment  trainers  (Puig, 
1972;  King,  1978).  Puig  (1972)  compared  two  groups  trained  on  either  actual 
equipment  or  a  training  device.  The  basis  of  the  comparison  was  the  amount 
of  time  each  group  spent  using  their  respective  training  equipment  when  both 
groups  achieved  approximately  the  same  level  of  proficiency  on  a  transfer 
task.  The  actual  equipment  cost  $6.00  per  hour  while  the  training  device 
cost  only  $2.00  per  hour,  so  Puig's  recommendation  was  to  use  the  train¬ 
ing  device  in  the  POI.  King  (1978)  approaches  the  same  comparison  by  empha¬ 
sizing  the  total  front  end  cost  of  the  training  device  and  the  actual  equip¬ 
ment.  Both  of  these  comparisons  are  narrow  in  that  they  do  not  actually 
center  on  training  effectiveness  improvements,  flexibility  in  the  training 
device,  safety,  or  other  factors  that  can  and  perhaps  should  be  included  in 
the  assessment. 

Another  general  method  of  cost  analysis  is  firmly  based  in  the  transfer 
effectiveness  ratio  (TER)  discussed  above.  As  Provenmire  and  Roscoe  (1973) 
pointed  out,  the  selection  of  training  devices  must  be  based  on  cost  ef¬ 
fectiveness,  which  in  turn  is  based  on  a  comparison  of  training  effectiveness. 
The  TER  allows  a  direct  comparison  in  terms  of  time  trial  costs.  The  TER 
demonstrates  how  much  can  be  saved  by  using  a  less  costly  training  device 
for  some  training  trials  rather  than  using  more  costly  actual  equipment 
trainers  for  all  of  training.  This  method  allows  training  device  assessments 
to  support  actual  cost  tradeoffs  while  maintaining  an  adequate  level  of 
proficiency  assessment  (Holman,  1979;  Bickley,  1980).  In  addition,  the 
use  of  incremental  transfer  effectiveness  ratios  (Roscoe,  1971;  Roscoe, 

1972)  based  on  varying  amounts  of  training  device  and  comparison  device 
utilization  allows  determination  of  the  most  effective  trade-off  point  for 
transfer  to  the  actual  equipment  (Provenmire  &  Roscoe,  1973;  Bickley,  1980). 
This  method  therefore  affords  the  opportunity  to  minimize  cost  while  maximiz¬ 
ing  transfer  (Fink  &  Shriver,  1978). 

Some  authors  (Lawrence,  1954;  Micheli,  1972)  argue  that  cost  effective¬ 
ness  can  and  should  be  extended  beyond  the  simple  comparison  of  equipment 
costs  to  other,  perhaps  more  important  areas.  Micheli  (1972)  argues  that 
the  possibility  of  variability  in  training,  the  training  control  differences, 
and  safety  factors  that  the  training  device  may  afford  should  be  entered  into 
the  overall  cost  analysis.  Lawrence  (1954)  argues  that  the  total  cost  of 
training  should  be  the  major  determinant  for  overall  training  program  evalua¬ 
tion.  The  measurements  and  cost  comparisons  should  also  be  based  on  super¬ 
visory  costs,  on-the-job  training  costs  over  the  entire  maintenance  career, 
equipment  and  personnel  damage  and  attendant  costs,  etc.  in  order  to  analyze 
the  complete  system  and  all  its  integral  parts.  Allbee  and  Semple  (1981) 
developed  a  hierarchical  cost  model  which  relates  training  costs  to  the  cost 
interests  of  various  levels  of  Air  Force  management.  They  recommend  that 
their  basic  model  be  expanded  to  include  more  subjective  parameters  such  as 
flight  safety  and  force  readiness.  Knerr  and  Matlick  (1980)  reviewed 
several  methods  for  applying  cost  and  training  effectiveness  analyses  at 
various  points  in  the  Army's  Life  Cycle  System  Management  Model  (LCSMM) .  A 
computerized  cost  effectiveness  model  was  developed  by  Marcus  et  al.  (1980). 
Further  research  to  validate  and  improve  the  applicability  of  these  models  is 
needed.  In  addition,  an  effort  to  synthesize  the  best  aspects  of  each  model 
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and  create  a  generalized  cost  effectiveness  analysis  methodology  would  greatly 
simplify  the  process  and  facilitate  its  use  in  the  Army  and  the  other  services. 
A  comprehensive  cost  effectiveness  analysis  methodology  will  afford  the 
opportunity  for  decision  makers  to  make  tradeoffs  between  the  costs  of  training 
devices  and  their  training  effectiveness. 

Most  training  effectiveness  studies  are  conducted  in  the  school  shortly 
following  training.  These  studies  tell  us  little  or  nothing  about  how  well 
the  trainee  does  in  the  field,  both  immediately  and  over  the  entire  career. 

In  addition,  there  are  very  few  adequate  cost  effectiveness  surveys  performed 
to  ensure  that  both  short-term  and  long-term  cost/proficiency  tradeoffs  are 
being  balanced.  Research  and  formalized  follow-up  testing  could  do  much  to 
provide  these  data.  A  system  of  follow-up  testing  procedures  would  require 
close  coordination  between  TRADOC  and  FORSCOM.  At  this  time  it  would  seem 
appropriate  to  conduct  a  feasibility  study  on  the  development  of  such 
coordination. 

Resource  Requirements.  An  incredible  amount  of  material  and  personnel 
resources  are  necessary  to  conduct  research  on  all  of  the  issues  described 
above.  The  problem  is  that  no  single  Army  agency  possesses  all  of  these  re¬ 
sources.  Additionally,  expertise  is  required  in  many  areas  from  test  design 
and  data  interpretation  to  weapon  operations  and  maintenance  procedures. 
Furthermore,  it  is  essential  that  the  ISD  process  make  the  best  use  of  sub¬ 
ject  matter  experts  from  all  fields.  One  approach  to  dealing  with  resource 
problems  would  consist  of  an  analytical  study  pinpointing  the  required  re¬ 
sources,  both  in  terms  of  technical  expertise  and  logistical  support,  at 
each  point  in  the  ISD  process.  This  effort  would  be  a  first  step  in  the 
establishment  of  resources  and  coordination  of  efforts  for  performing  research 
into  all  of  the  issues  raised  in  this  article.  Another  approach,  perhaps 
more  realistic  in  terms  of  scope  and  resource  utilization,  would  be  to  organize 
existing  data  and  apply  the  ISD  process  to  a  particular  training  program  in 
order  to  identify  and  prioritize  research  issues  for  that  program.  Ttiis 
effort  would  serve  as  a  trial  run  for  the  procedures  that  would  eventually  be 
required  for  Army-wide  training  programs.  Such  a  smaller  scale  effort  could 
help  in  determining  the  feasibility  of  an  Army-wide  effort  and  would  be  within 
the  resource  constraints  of  a  single  Army  agency. 


SUMMARY  AND  RECOMMENDATIONS 

In  this  paper  we  have  argued  that  a  training  device  research  issues  data 
base  and  training  device  design  guidance  system  should  be  a  primary  goal  in 
the  Army  training  community.  It  is  our  belief  that  informed  decisions  on  the 
design  and  implementation  of  training  devices  and  training  programs  must  be 
based  on  the  best  available  empirically  derived  information  on  the  training 
effects  of  various  training  system  variables.  In  order  to  provide  the 
conceptual  organization  for  this  training  system  research  issues  data  base, 
we  have  accumulated  some  of  the  existing  data  and  organized  the  information 
around  important  development  and  research  questions.  Our  approach  emphasizes 
the  use  of  existing  data  for  short  term  guidance  but  the  iterative  development 
of  improved  guidance  as  new  data  are  developed.  A  well  organized  data  base 
will  facilitate  both  of  these  goals  and  an  automated  expert  system  can  help 
insure  that  training  system  design  guidance  is  delivered  to  users  in  a  form 
that  will  be  of  immediate  value. 
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In  the  first  section  of  the  paper,  a  tentative  plan  for  the  accumulation 
and  distribution  of  the  data  derived  from  the  study  of  training  system  re¬ 
search  issues  was  presented.  This  plan  recommends  the  establishment  of  an 
iterative  data  base,  which  would  be  updated  as  newer,  more  valid  and  reliable 
data  are  developed.  This  data  base  is  proposed  as  the  foundation  for  user- 
oriented  training  system  design  guidance.  Such  guidance  would  be  organized 
around  user-generated  questions  and  would  translate  raw  data  into  guidance 
statements  which  would  be  of  direct  value  to  training  system  designers.  This 
guidance  could  take  the  initial  form  of  a  guidebook  but  the  authors  feel 
that  automation  in  an  expert  system  format  would  be  more  advantageous  over 
the  long  run . 

The  second  section  is  a  presentation  of  the  basic  instructional  system 
design  (ISD)  progression  in  terms  of  the  major  areas  and  relationships  in  the 
cycle.  The  emphasis  is  that  no  single  research  issue  connected  to  a  single 
stage  in  the  ISD  process  stands  alone.  The  systemic  nature  of  the  ISD  process 
forces  training  program  developers,  administrators,  and  researchers  to  view 
training  system  issues  as  completely  interactive  and  interdependent.  It  is 
the  authors'  opinion  that  very  little  progress  can  be  made  in  the  investigation 
of  training  system  and  training  device  issues  if  one  maintains  an  isolationist 
view  of  these  issues. 

In  the  third  major  section  of  the  paper,  we  presented  some  of  the 
elaborations  of  the  main  ISD  research  issues  which  we  feel  must  be  studied 
if  instructional  programs  are  to  be  designed  for  maximum  efficiency  and 
effectiveness.  Each  research  issue  was  discussed  not  only  as  it  relates  to 
training  device  design,  but  also  as  it  interacts  with  other  issues  and  other 
aspects  of  ISD.  Existing  data  from  previous  research  were  also  reviewed  in 
this  section.  Table  7  provides  a  summary  of  all  the  research  questions 
discussed  in  this  section. 

The  authors  have  not  attempted,  in  this  paper,  to  assign  priorities  to 
the  research  issues.  ARI  has,  however,  several  ongoing  efforts  in  the  area 
of  training  device  fidelity  which  will  provide  the  first  entries  into  the 
data  base.  One  experiment  (Baum,  Riedel  &  Hays,  in  press)  investigated  the 
effects  of  simulator  fidelity  in  a  mechanical  maintenance  task.  This  experiment 
is  currently  being  replicated  at  ARI.  An  additional  series  of  experiments  is 
ongoing  at  the  George  Mason  University,  Fairfax,  Virginia  (Allen  &  Hays, 

1983).  In  this  series  of  experiments  the  effects  of  simulator  fidelity  are 
being  investigated  in  the  context  of  an  electro-mechanical  troubleshooting 
task.  Several  interactive  variables,  such  as  trainee  aptitude,  task  difficulty 
and  instructional  strategy  will  also  be  investigated  in  this  effort.  A  third 
effort  will  produce  an  annotated  bibliography  of  over  150  training  device 
effectiveness  articles.  The  bibliography  is  being  formatted  for  direct  entry 
into  the  data  base.  As  soon  as  these  first  entries  are  incorporated  into  a 
prototype  data  base  the  data  themselves  can  be  used  to  help  researchers  assign 
priorities  to  the  issues  discussed  in  this  paper. 


Table  7 


I 

fc 


Summary  of  Training  System  Issues  Requiring  Empirical  Investigation 


Area  of  Interest 


Research  Questions 


I.  Training  Needs  Assessment 

A.  Task  Analysis  1.  How  are  task  analyses  currently 

conducted  in  the  Army  and  the 
other  services? 

2.  Can  a  standardized  task  analysis 
format  and  methodology  be  developed? 
What  are  its  characteristics? 

3 .  Can  a  workable  taxonomy  of  tasks 
be  developed?  What  are  its 
characte ristics? 


B.  Media  Selection  1 .  How  are  media  selection  models 

currently  used  in  the  Army  and  the 
other  services? 

2.  Which,  if  any,  existing  media  selec¬ 
tion  models  are  valid?  What  modifi¬ 
cations  are  required  to  increase 
their  validity? 

3.  Can  a  media  selection  model  be  de¬ 
veloped  which  includes  specifica¬ 
tions  for  the  characteristics  of 
the  instructional  medium  selected? 

4.  What  are  the  training  effects  of 
various  media  mixes? 

5.  What  is  the  appropriate  instruc¬ 
tional  medium  for  use  at  each  stage 
of  training? 

6.  How  does  the  interaction  of  media 
and  media  mixes  with  other  training 
system  variables  affect  training? 


C.  Trainee  Characteristics  1.  What  are  the  appropriate  training 

device  characteristics  and  train¬ 
ing  strategies  for  each  type  of 
trainee? 

2.  What  are  the  enabling  skills 
required  for  each  type  of  training 
device? 

3 .  How  can  trainee  motivation  be  manipu¬ 
lated  to  improve  training? 

4.  What  are  the  effects  of  learning 
set  on  training? 

5.  Are  there  neurological  differences 
among  trainees  that  affect  training 
outcomes?  How  can  training  ef¬ 
fectiveness  be  improved  by  taking 
these  neurological  differences 
into  account? 
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Table  7  (continued) 


Area  of  Interest 

D.  Training  Strategies  1. 

2. 

3. 

4. 

5. 

6. 

II.  Training  Device  Design 

A.  Training  Device  Fidelity  1. 

2. 


Research  Questions 

When  is  a  general  rather  than  a  spe¬ 
cific  training  strategy  appropriate? 
When  should  generic  rather  than 
specific  tasks  be  trained? 

When  is  part-task  rather  than  whole- 
task  training  more  appropriate? 

When  is  it  more  training  effective 
to  use  lock-step  rather  than  self- 
paced  training?  Can  a  mix  of  the 
two  types  be  training  effective? 

When  is  it  more  training  effective 
to  use  trial  and  error  learning 
rather  than  error  free  learning? 

How  are  current  learning  guidelines 
used?  Are  current  guidelines  valid 
and  user-oriented?  How  might  their 
use  be  improved? 


What  are  the  relative  training  ef¬ 
fects  of  various  levels  of  physical 
and  functional  fidelity? 

How  do  the  two  aspects  of  fidelity 
interact  with  other  training  sys¬ 
tem  variables? 


B.  Instructional  Features 


C.  Training  Device 
Architecture 


1 .  What  are  the  training  effects  of 
each  specific  instructional  feature 
that  can  be  included  in  a  training 
device? 

2.  When  should  training  feedback  be 
augmented  and  by  what  means? 

3.  When  and  how  should  fading  be  used 
in  training? 

1 .  When  is  it  more  training  effective 
to  use  2D  rather  than  3D  for  hands- 
on  practice? 

2.  What  is  the  best  mix  of  student  and 
instructor  stations  for  training 
specific  tasks  with  different  train¬ 
ing  devices  and  training  strategies? 

3.  When  are  each  of  the  different 
types  of  trainee-device  interac¬ 
tion  most  training  effective? 
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Table  7  (continued) 


III. 


IV. 


Area  of  Interest 


Research  Questions 


Training  Device  Implementation 
A.  User  Acceptance  1 


2 


B.  Training  Device  1 

Integration 

2 


C.  Noninstitutional  Use  of  1 

Training  Devices 

2 

3 


4 


5 


What  are  the  factors  that  make  a 
training  device  acceptable  to 
instructors?  to  students? 

What  are  the  effects  of  nonacceptance 
of  training  devices? 

How  can  training  devices  be  most 
easily  integrated  into  POIs? 

What  are  the  training  effects  of 
poor  device-POI  integration? 

What  are  the  candidate  noninstitu¬ 
tional  uses  for  training  devices? 

How  do  specific  tasks  change  when 
in  a  noninstitutional  context? 

What  are  the  special  problems  with 
device  acceptance  and  device  inte¬ 
gration  in  a  noninstitutional 
context? 

When  and  what  types  of  refresher 
training  could  be  delivered  with 
noninstitutional  training  devices? 

How  would  devices  be  evaluated  in 
a  noninstitutional  context? 


Training  Effectiveness 
Assessment 

A.  Evaluation  Measurement  1 


2 


3 


4 


5 


6 


What  are  the  relative  values  of 
process  and  outcome  measures?  When 
is  the  use  of  one  rather  than  the 
other  more  appropriate? 

What  are  the  relative  values  of 
proximal  and  distal  measures? 
when  is  the  use  of  one  rather 
than  the  other  more  appropriate? 

What  is  the  appropriate  mix  of 
objective  and  subjective  criteria 
in  training  effectiveness  evaluation? 
What  are  the  relative  values  of 
criterion  and  norm- referenced  per¬ 
formance  standards?  When  should 
one  be  used  rather  than  the  other? 
What  is  the  best  mix  of  the  above 
types  of  measures  for  specific 
contexts? 

How  can  training  effectiveness 
measures  be  made  more  valid? 
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Table  7  (continued) 


Area  of  Interest 


Research 


jestions 


B.  Types  of  Training 

Effectiveness  Assessment  1 .  How  does  the  Army  develop  and  use 

proficiency  tests? 

2.  Which  types  of  proficiency  measures 
provide  the  best  inputs  for  train¬ 
ing  effectiveness  evaluation? 


Training  Effectiveness 
Assessment  (continued) 

B.  Types  of  Training 

Effectiveness  Assessment  3.  How  do  the  data  from  other  train- 
( continued)  ing  effectiveness  measures  corre¬ 

late  with  transfer  of  training 
data? 

4.  How  can  cost  effectiveness  and 
training  effectiveness  data  be 
best  used  to  evaluate  training? 
What  is  the  validity  of  existing 
cost  effectiveness  assessment 
models?  How  can  their  applica¬ 
tion  be  enhanced? 

5.  What  is  the  validity  of  existing 
analytic  models  for  assessing 
training  effectiveness?  How  might 
such  models  be  improved? 

6.  How  can  coordination  be  estab¬ 
lished  so  long-term  retention  data 
can  be  used  to  evaluate  training 
ef  f ectiveness? 


C.  Resource  Requirements  1.  What  are  the  required  resources 

to  adequately  assess  training 
effectiveness? 

2.  From  which  agencies  within  the 
Army  and  the  other  services  can 
the  necessary  resources  to  assess 
training  effectiveness  be  obtained? 

3.  How  can  the  required  coordination 
between  the  above  agencies  be 
facilitated? 
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1 

I 

I 

If  training  programs  are  to  keep  pace  with  rapidly  evolving  technologies 
and  provide  the  most  cost  effective  training  possible,  their  design  and  imple¬ 
mentation  can  no  longer  rely  on  merely  intuitive  approaches.  It  behooves 
instructional  system  designers  to  use  the  best  available  empirical  data  to 
aid  in  the  design  of  training  devices  and  training  systems.  An  expert  system 
that  is  grounded  in  empirical  data  on  each  and  every  research  question  dis¬ 
cussed  in  this  paper  would  be  an  invaluable  aid  to  the  training  community. 

The  use  of  such  an  expert  system  would  improve  both  the  cost  and  the  train¬ 
ing  effectiveness  of  instructional  delivery  systems. 


I 
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Appendix  A:  Types  of  Task  Analyses 


There  have  been  numerous  approaches  to  the  analysis  of  tasks.  Several 
researchers  have  attempted  to  classify  these  approaches.  One  of  the  most  widely 
used  classifications  of  types  of  task  analyses  divides  them  into  four  conceptual 
bases  (Wheaton,  1968;  Fleishman,  1975;  Fleishman,  1977): 

1.  The  Behavior  Description  Approach 

2.  The  Behavior  Requirements  Approaches 

3.  The  Abilities  Requirements  Approach 


i 
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4.  The  Task  Characteristics  Approach 

The  (1)  behavior  description  approach  (McCormick,  Jeanneret  &  Mecham,  1972)  is 
based  upon  observations  and  descriptions  of  what  people  actually  do  while 
performing  a  task.  It  provides  an  observer's  view  of  the  actions  of  a  task 
performer.  An  example  of  a  behavior  description  might  be  "pulls  lever  until 
pressure  warning  light  goes  out."  The  (2)  behavior  requirements  approach 
(Gagne,  1962;  R.  Miller,  1962;  Annette  &  Duncan,  1967),  emphasizes  the  cataloging 
of  behaviors  which  are  assumed  to  be  required  in  order  to  achieve  criterion 
levels  of  performance.  This  approach  would,  for  example,  detail  how  quickly, 
with  how  much  force,  and  for  what  duration  the  above  lever  would  have  to  be 
pulled  in  order  to  adjust  the  pressure  to  its  desired  level.  The  (3)  abilities 
requirements  approach  (Fleishman  1977;  Theologus  £  Fleishman,  1971),  describes, 
contrasts,  and  compares  tasks  in  terms  of  the  abilities  that  are  conceived  as 
being  relatively  enduring  attributes  of  the  individual  performing  the  task. 

The  assumption  is  that  different  tasks  require  different  abilities.  In  our 
lever  pulling  example,  this  approach  would  focus  on  the  motor  skills  and  the 
perceptual  requirements  necessary  for  individuals  to  accomplish  the  lever  ad¬ 
justment.  Finally,  in  the  (4)  task  characteristics  approach  (Fleishman,  1972; 
Farina  £  Wheaton,  1971;  Hackman,  1970),  task  description  is  predicated  upon  a 
definition  that  treats  the  task  as  a  set  of  conditions  which  elicit  performance. 
The  assumption  is  that  tasks  may  be  described  and  differentiated  in  terms  of 
intrinsic  objective  properties  which  they  may  possess.  The  components  of  a 
task  (an  explicit  goal,  procedures,  input  stimuli,  responses  and  stimulus- 
response  relationships)  are  treated  as  categories  within  which  to  devise  task 
characteristics  or  descriptions. 

Besides  the  four  discussed  above,  two  additional  conceptual  bases  have 
been  described. 

5.  The  Phenomenological  Approach 

6.  The  Information-Theoretic  Approach 

The  (5)  phenomenological  approach  (Klein,  1977)  focuses  on  the  way  the  task  is 
experienced.  It  seeks  to  provide  a  holistic  understanding  of  the  system  in 
which  the  task  is  embedded.  In  the  (6)  information-theoretic  approach  (Levine 
&  Teichner,  1971),  the  task  is  conceived  as  a  transfer  of  information  between 
components  of  the  system  (man-machine,  machine-man,  man-man,  or  machine-machine). 
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Tasks  are  categorized  based  upon  the  constraints  on  information  flow  between 
components . 

These  six  approaches  to  task  analysis  are  alternative  ways  of  viewing  the 
process  of  task  analysis.  Each  has  a  different  goal  and  produces  a  different 
form  of  output.  The  distinctions  between  the  approaches  are  important  because 
by  choosing  one  approach  over  the  others,  we  are  likely  to  obtain  different 
results.  These  various  results  are  due  to  the  different  criteria  each  approach 
applies  to  the  analysis  of  the  task.  In  1969  Farina  made  a  statement  which  is 
probably  still  true  today.  He  stated  that  there  are  no  deliberate  eclectics  in 
the  field  of  behavior  description.  Each  researcher  seems  to  have  his/her  own 
purpose  and  chooses  the  approach  which  fits  the  purpose  most  closely.  Farina 
also  makes  the  important  point  that  performance  is  a  function  of  the  task,  the 
characteristics  of  the  operator,  and  the  characteristics  of  the  environment 
(P  =  f(T,0,E)).  While  behavioral  descriptors  focus  on  the  0  portion  of  the 
equation,  it  is  also  necessary  to  obtain  descriptors  for  the  T  and  E  portions. 
Table  A  lists  these  six  approaches  and  indicates  to  which  portion  of  the  above 
equation  they  most  closely  apply.  As  can  be  seen  from  Table  A,  each  type  of 
task  analysis,  while  not  necessarily  ignoring  all  other  areas,  does  have  a 
central  area  of  concern. 


Table  A 

Six  Approaches  to  Task  Analysis 
and  Their  Main  Areas  of  Concern 


Approach 

Main  Area  of  Concern 

1. 

Behavior  Description  Approach 

Operator 

2. 

Behavior  Requirements  Approach 

Task 

3. 

Abilities  Requirements  Approach 

Operator 

4. 

Task  Characteristics  Approach 

Task 

5. 

Phenomenological  Approach 

Operator 

6. 

Information-Theoretic  Approach 

Task/Environment 
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